GCC Middle and Back End API Reference
|
#include <profile-count.h>
Static Public Member Functions | |
static profile_count | zero () |
static profile_count | adjusted_zero () |
static profile_count | guessed_zero () |
static profile_count | one () |
static profile_count | uninitialized () |
static void | adjust_for_ipa_scaling (profile_count *num, profile_count *den) |
static profile_count | from_gcov_type (gcov_type v, profile_quality quality=PRECISE) |
static profile_count | stream_in (class lto_input_block *) |
Static Public Attributes | |
static const int | n_bits = 61 |
static const uint64_t | max_count = ((uint64_t) 1 << n_bits) - 2 |
Private Attributes | |
uint64_t UINT64_BIT_FIELD_ALIGN | m_val: n_bits |
enum profile_quality | m_quality: 3 |
Static Private Attributes | |
static const uint64_t | uninitialized_count = ((uint64_t) 1 << n_bits) - 1 |
Main data type to hold profile counters in GCC. Profile counts originate either from profile feedback, static profile estimation or both. We do not perform whole program profile propagation and thus profile estimation counters are often local to function, while counters from profile feedback (or special cases of profile estimation) can be used inter-procedurally. There are 3 basic types 1) local counters which are result of intra-procedural static profile estimation. 2) ipa counters which are result of profile feedback or special case of static profile estimation (such as in function main). 3) counters which counts as 0 inter-procedurally (because given function was never run in train feedback) but they hold local static profile estimate. Counters of type 1 and 3 cannot be mixed with counters of different type within operation (because whole function should use one type of counter) with exception that global zero mix in most operations where outcome is well defined. To take local counter and use it inter-procedurally use ipa member function which strips information irrelevant at the inter-procedural level. Counters are 61bit integers representing number of executions during the train run or normalized frequency within the function. As the profile is maintained during the compilation, many adjustments are made. Not all transformations can be made precisely, most importantly when code is being duplicated. It also may happen that part of CFG has profile counts known while other do not - for example when LTO optimizing partly profiled program or when profile was lost due to COMDAT merging. For this reason profile_count tracks more information than just unsigned integer and it is also ready for profile mismatches. The API of this data type represent operations that are natural on profile counts - sum, difference and operation with scales and probabilities. All operations are safe by never getting negative counts and they do end up in uninitialized scale if any of the parameters is uninitialized. All comparisons that are three state and handling of probabilities. Thus a < b is not equal to !(a >= b). The following pre-defined counts are available: profile_count::zero () for code that is known to execute zero times at runtime (this can be detected statically i.e. for paths leading to abort (); profile_count::one () for code that is known to execute once (such as main () function profile_count::uninitialized () for unknown execution count.
|
static |
We want to scale profile across function boundary from NUM to DEN. Take care of the side case when DEN is zeros. We still want to behave sanely here which means - scale to profile_count::zero () if NUM is profile_count::zero - do not affect anything if NUM == DEN - preserve counter value but adjust quality in other cases.
References force_nonzero(), and zero().
Referenced by cgraph_edge::clone(), copy_bb(), copy_cfg_body(), execute_fixup_cfg(), initialize_cfun(), inline_transform(), scale_ipa_profile_for_fn(), update_counts_for_self_gen_clones(), update_noncloned_counts(), and update_profiling_info().
|
inlinestatic |
References ADJUSTED, m_quality, and m_val.
Referenced by ipa(), and update_profiling_info().
|
inline |
References AFDO, and m_quality.
Referenced by autofdo::afdo_annotate_cfg(), autofdo::afdo_calculate_branch_prob(), afdo_callsite_hot_enough_for_early_inline(), autofdo::afdo_propagate_edge(), and autofdo::afdo_set_bb_count().
|
inline |
References ADJUSTED, gcc_checking_assert, initialized_p(), m_quality, m_val, MIN, REG_BR_PROB_BASE, safe_scale_64bit(), and uninitialized().
Referenced by compute_path_counts(), copy_bb(), tree_switch_conversion::switch_decision_tree::do_jump_if_equal(), duplicate_loop_body_to_header_edge(), tree_switch_conversion::switch_decision_tree::emit_case_nodes(), tree_switch_conversion::switch_decision_tree::emit_cmp_and_jump_insns(), execute_sm_if_changed(), expand_complex_div_wide(), fix_loop_bb_probability(), gimple_ic(), ipa_profile(), vect_loop_versioning(), and vect_transform_loop().
|
inline |
|
inline |
References ADJUSTED, gcc_checking_assert, initialized_p(), m_quality, m_val, max_count, MIN, safe_scale_64bit(), and uninitialized().
Referenced by copy_bb(), copy_edges_for_bb(), cgraph_node::dump(), duplicate_loop_body_to_header_edge(), execute_fixup_cfg(), inline_transform(), input_bb(), ipa_devirt(), ipa_make_edge_direct_to_target(), ipa_merge_profiles(), ipcp_cloning_candidate_p(), merge_profile_summaries(), operator*(), operator*=(), operator/(), operator/=(), scale_dominated_blocks_in_loop(), scale_ipa_profile_for_fn(), scale_strictly_dominated_blocks(), try_shrink_wrapping(), update_bb_profile_for_threading(), update_noncloned_counts(), update_profiling_info(), and update_specialized_profile().
|
inline |
References ADJUSTED, gcc_checking_assert, GUESSED, initialized_p(), ipa(), ipa_p(), m_quality, m_val, MAX, max_count, MIN, safe_scale_64bit(), uninitialized(), and zero().
profile_count profile_count::combine_with_ipa_count | ( | profile_count | ipa | ) |
THIS is a count of bb which is known to be executed IPA times. Combine this information into bb counter. This means returning IPA if it is nonzero, not changing anything if IPA is uninitialized and if IPA is zero, turning THIS into corresponding local profile with global0.
References global0(), global0adjusted(), initialized_p(), ipa(), and zero().
Referenced by adjust_clone_incoming_counts(), combine_with_ipa_count_within(), and update_profiling_info().
profile_count profile_count::combine_with_ipa_count_within | ( | profile_count | ipa, |
profile_count | ipa2 ) |
Sae as profile_count::combine_with_ipa_count but within function with count IPA2.
References combine_with_ipa_count(), compatible_p(), gcc_checking_assert, initialized_p(), and ipa().
|
inline |
References initialized_p(), ipa(), ipa_p(), nonzero_p(), and zero().
Referenced by adjust_clone_incoming_counts(), combine_with_ipa_count_within(), differs_from_p(), max(), operator+(), operator+=(), operator-(), operator-=(), operator<(), operator<=(), operator>(), operator>=(), probability_in(), to_cgraph_frequency(), to_sreal_scale(), and cgraph_node::verify_node().
void profile_count::debug | ( | ) | const |
bool profile_count::differs_from_p | ( | profile_count | other | ) | const |
Return true if THIS differs from OTHER; tolerate small differences.
References compatible_p(), gcc_checking_assert, initialized_p(), m_val, and safe_scale_64bit().
Referenced by check_bb_profile(), expected_loop_iterations_by_profile(), profile_record_check_consistency(), rebuild_frequencies(), update_loop_exit_probability_scale_dom_bbs(), and cgraph_node::verify_node().
Dump THIS to F.
References function::cfg, count, ENTRY_BLOCK_PTR_FOR_FN, initialized_p(), m_quality, m_val, PRId64, profile_quality_display_names, and to_sreal_scale().
Referenced by check_bb_profile(), compute_alignments(), consider_split(), debug(), cgraph_node::dump(), dump_basic_block_info(), dump_bb_info(), dump_prediction(), dump_profile_updates(), find_traces(), good_cloning_opportunity_p(), inline_transform(), ipa_merge_profiles(), param_change_prob(), print_loop_info(), cgraph_edge::redirect_call_stmt_to_callee(), update_loop_exit_probability_scale_dom_bbs(), update_profiling_info(), update_specialized_profile(), and cgraph_node::verify_node().
|
inline |
References ADJUSTED, initialized_p(), m_quality, m_val, and MIN.
Referenced by adjust_for_ipa_scaling().
|
static |
The profiling runtime uses gcov_type, which is usually 64bit integer. Conversions back and forth are used to read the coverage and get it into internal representation.
References dump_file, gcc_checking_assert, m_quality, m_val, max_count, MIN, PRId64, and quality().
Referenced by autofdo::afdo_annotate_cfg(), afdo_callsite_hot_enough_for_early_inline(), autofdo::afdo_set_bb_count(), compute_branch_probabilities(), cgraph_node::dump(), estimate_bb_frequencies(), expand_thunk(), gimple_divmod_fixed_value(), gimple_mod_pow2(), gimple_mod_subtract(), gimple_stringop_fixed_value(), one(), read_thunk_profile(), and zero().
|
inline |
References GUESSED_GLOBAL0, initialized_p(), and m_quality.
Referenced by combine_with_ipa_count(), compute_branch_probabilities(), and update_profiling_info().
|
inline |
References GUESSED_GLOBAL0_ADJUSTED, initialized_p(), and m_quality.
Referenced by combine_with_ipa_count().
|
inline |
References GUESSED, m_quality, and MIN.
Referenced by update_bb_profile_for_threading().
|
inline |
References GUESSED_LOCAL, initialized_p(), and m_quality.
Referenced by drop_profile(), expand_thunk(), lenient_count_portion_handling(), localize_profile(), and update_profiling_info().
|
inlinestatic |
References GUESSED, m_quality, and m_val.
Referenced by compute_branch_probabilities().
|
inline |
References m_val, and uninitialized_count.
Referenced by apply_probability(), apply_probability(), apply_scale(), apply_scale(), better_edge_p(), combine_with_ipa_count(), combine_with_ipa_count_within(), compatible_p(), compute_alignments(), differs_from_p(), dump(), dump_basic_block_info(), dump_bb_for_graph(), dump_bb_info(), dump_function_to_file(), dump_gimple_bb_header(), dump_inline_stats(), dump_overall_stats(), dump_prediction(), back_jt_path_registry::duplicate_thread_path(), edge_badness(), execute_fixup_cfg(), expand_thunk(), expected_loop_iterations_by_profile(), expensive_function_p(), expr_expected_value_1(), find_best_predecessor(), force_nonzero(), gather_caller_stats(), get_info_about_necessary_edges(), gimple_account_profile_record(), gimple_duplicate_sese_tail(), gimple_verify_flow_info(), global0(), global0adjusted(), guessed_local(), handle_missing_profiles(), inline_transform(), ipa_merge_profiles(), ipa_p(), ipa_profile(), ipa_profile_generate_summary(), ipcp_cloning_candidate_p(), ipcp_propagate_stage(), max(), cgraph_edge::maybe_hot_p(), merge(), nonzero_p(), operator+(), operator+=(), operator-(), operator-=(), operator<(), operator<(), operator<=(), operator<=(), operator>(), operator>(), operator>=(), operator>=(), perhaps_add_new_callers(), probability_in(), profile_record_check_consistency(), rebuild_frequencies(), recursive_inlining(), replace_block_by(), report_unroll(), rtl_account_profile_record(), sanitize_hot_paths(), to_cgraph_frequency(), to_frequency(), to_gcov_type(), to_sreal_scale(), try_unroll_loop_completely(), update_bb_profile_for_threading(), update_loop_exit_probability_scale_dom_bbs(), update_profile(), verify_flow_info(), very_large_p(), and want_inline_small_function_p().
|
inline |
References adjusted_zero(), GUESSED_GLOBAL0, GUESSED_GLOBAL0_ADJUSTED, m_quality, uninitialized(), and zero().
Referenced by adjust_clone_incoming_counts(), analyze_clone_icoming_counts(), apply_scale(), branch_prob(), check_counter(), cgraph_edge::clone(), combine_with_ipa_count(), combine_with_ipa_count_within(), compatible_p(), cgraph_node::create_clone(), decide_about_value(), dump_inline_stats(), dump_overall_stats(), edge_badness(), estimate_bb_frequencies(), gather_caller_stats(), gather_count_of_non_rec_edges(), get_info_about_necessary_edges(), gimple_account_profile_record(), handle_missing_profiles(), input_bb(), ipa_devirt(), ipa_merge_profiles(), ipa_profile_generate_summary(), ipcp_cloning_candidate_p(), ipcp_propagate_stage(), lenient_count_portion_handling(), max(), merge_profile_summaries(), operator<(), operator<=(), operator>(), operator>=(), perhaps_add_new_callers(), profile_record_check_consistency(), rtl_account_profile_record(), update_counts_for_self_gen_clones(), update_max_bb_count(), update_profiling_info(), update_specialized_profile(), and want_inline_small_function_p().
|
inline |
References GUESSED_GLOBAL0, initialized_p(), and m_quality.
Referenced by apply_scale(), compatible_p(), do_estimate_edge_time(), expand_thunk(), inline_small_functions(), ipa_profile_generate_summary(), lenient_count_portion_handling(), operator<(), operator<=(), operator>(), operator>=(), and cgraph_node::verify_node().
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), ipa(), m_quality, m_val, nonzero_p(), and zero().
Referenced by edge_order(), and update_max_bb_count().
|
inline |
References initialized_p(), and m_quality.
|
inline |
References initialized_p(), and m_val.
Referenced by autofdo::afdo_calculate_branch_prob(), branch_prob(), profile_probability::combine_with_count(), compatible_p(), decide_about_value(), duplicate_loop_body_to_header_edge(), edge_badness(), execute_sm_if_changed(), expected_loop_iterations_by_profile(), expr_expected_value_1(), fold_loop_internal_call(), good_cloning_opportunity_p(), handle_missing_profiles(), hoist_guard(), input_bb(), ipa_merge_profiles(), lenient_count_portion_handling(), max(), merge_profile_summaries(), param_change_prob(), perhaps_add_new_callers(), recursive_inlining(), scale_bbs_frequencies_profile_count(), scale_dominated_blocks_in_loop(), scale_profile_for_vect_loop(), scale_strictly_dominated_blocks(), speculation_useful_p(), try_shrink_wrapping(), update_bb_profile_for_threading(), update_max_bb_count(), update_profile_after_ch(), update_profiling_info(), vect_do_peeling(), cgraph_node::verify_node(), and want_inline_small_function_p().
|
inline |
References ADJUSTED, and m_quality.
Referenced by want_merge_blocks_p().
|
inlinestatic |
References from_gcov_type().
Referenced by expand_call_inline().
|
inline |
References apply_scale().
|
inline |
References apply_scale().
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_quality, m_val, max_count, MIN, uninitialized(), and zero().
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_quality, m_val, max_count, MIN, uninitialized(), and zero().
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_quality, m_val, MIN, uninitialized(), and zero().
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_quality, m_val, MIN, uninitialized(), and zero().
|
inline |
References apply_scale().
|
inline |
References apply_scale().
References gcc_checking_assert, initialized_p(), ipa(), ipa_p(), and m_val.
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_val, and zero().
References gcc_checking_assert, initialized_p(), ipa(), ipa_p(), and m_val.
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_val, and zero().
|
inline |
References gcc_checking_assert, initialized_p(), ipa(), ipa_p(), and m_val.
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_val, and zero().
References gcc_checking_assert, initialized_p(), ipa(), ipa_p(), and m_val.
|
inline |
References compatible_p(), gcc_checking_assert, initialized_p(), m_val, and zero().
|
inline |
References m_quality, and PRECISE.
Referenced by dump_prediction(), and find_rarely_executed_basic_blocks_and_crossing_edges().
|
inline |
References ADJUSTED, profile_probability::always(), compatible_p(), gcc_checking_assert, GUESSED, initialized_p(), m_quality, profile_probability::m_quality, m_val, profile_probability::m_val, MAX, profile_probability::max_probability, MIN, profile_probability::never(), PRECISE, RDIV, profile_probability::uninitialized(), and zero().
Referenced by profile_probability::combine_with_count(), compute_path_counts(), copy_bb(), duplicate_loop_body_to_header_edge(), execute_sm_if_changed(), good_cloning_opportunity_p(), ipa_merge_profiles(), cgraph_edge::redirect_call_stmt_to_callee(), scale_profile_for_vect_loop(), to_frequency(), update_loop_exit_probability_scale_dom_bbs(), update_profile(), update_profile_after_ch(), and vect_do_peeling().
|
inline |
References m_quality.
Referenced by dump_function_to_file(), dump_gimple_bb_header(), and from_gcov_type().
|
inline |
References ADJUSTED, and m_quality.
Referenced by expected_loop_iterations_by_profile().
|
static |
Stream THIS from IB.
References m_quality, m_val, streamer_read_gcov_count(), and streamer_read_uhwi().
Referenced by input_bb(), input_edge(), and input_node().
void profile_count::stream_out | ( | struct lto_output_stream * | ob | ) |
Stream THIS to OB.
References m_quality, m_val, streamer_write_gcov_count_stream(), and streamer_write_uhwi_stream().
void profile_count::stream_out | ( | struct output_block * | ob | ) |
Stream THIS to OB.
References m_quality, m_val, streamer_write_gcov_count(), and streamer_write_uhwi().
Referenced by lto_output_node(), and output_bb().
int profile_count::to_cgraph_frequency | ( | profile_count | entry_bb_count | ) | const |
Return count as frequency within FUN scaled in range 0 to CGRAPH_FREQ_MAX where CGRAPH_FREQ_BASE means that count equals to entry block count. Used for legacy code and should not be used anymore.
References CGRAPH_FREQ_BASE, CGRAPH_FREQ_MAX, compatible_p(), gcc_checking_assert, initialized_p(), m_val, MAX, MIN, safe_scale_64bit(), and zero().
Referenced by compute_call_stmt_bb_frequency().
int profile_count::to_frequency | ( | struct function * | fun | ) | const |
Return count as frequency within FUN scaled in range 0 to REG_FREQ_MAX Used for legacy code and should not be used anymore.
References BB_FREQ_MAX, function::cfg, control_flow_graph::count_max, gcc_assert, initialized_p(), profile_probability::initialized_p(), probability_in(), REG_BR_PROB_BASE, STATIC_ASSERT, profile_probability::to_reg_br_prob_base(), and zero().
Referenced by bb_to_key(), coalesce_cost_bb(), determine_scaling_factor(), find_best_predecessor(), find_trace(), loop_compare_func(), mark_loops_for_removal(), place_prologue_for_one_component(), and tail_duplicate().
|
inline |
References gcc_checking_assert, initialized_p(), and m_val.
Referenced by check_counter(), dump_bb_for_graph(), dump_inline_stats(), dump_overall_stats(), dump_prediction(), edge_badness(), eliminate_partially_redundant_load(), gimple_account_profile_record(), ipa_devirt(), ipa_profile_generate_summary(), profile_record_check_consistency(), recursive_inlining(), report_unroll(), rtl_account_profile_record(), and try_unroll_loop_completely().
sreal profile_count::to_sreal_scale | ( | profile_count | in, |
bool * | known = NULL ) const |
Return THIS/IN as sreal value.
References compatible_p(), gcc_checking_assert, initialized_p(), m_val, and zero().
Referenced by analyze_function_body(), average_num_loop_insns(), compute_branch_probabilities(), dump(), execute_split_functions(), gimple_account_profile_record(), param_change_prob(), profile_record_check_consistency(), rtl_account_profile_record(), and vect_slp_node_weight().
|
inlinestatic |
References GUESSED_LOCAL, m_quality, m_val, and uninitialized_count.
Referenced by autofdo::afdo_indirect_call(), alloc_block(), apply_probability(), apply_probability(), apply_scale(), apply_scale(), create_block_for_threading(), doloop_modify(), drop_profile(), duplicate_block(), estimate_bb_frequencies(), find_bb_boundaries(), find_traces_1_round(), gimple_duplicate_sese_tail(), init_flow(), inline_small_functions(), inline_transform(), input_node(), ipa(), ipa_cp_cc_finalize(), ipcp_propagate_stage(), operator+(), operator+=(), operator-(), operator-=(), rebuild_frequencies(), rotate_loop(), sanitize_hot_paths(), scale_loop_profile(), update_exit_probability_after_unrolling(), update_max_bb_count(), and update_profile_for_new_sub_basic_block().
|
inline |
References m_val.
Referenced by dump_function_to_file(), and dump_gimple_bb_header().
|
inline |
References gcc_checking_assert, GUESSED_LOCAL, m_quality, m_val, uninitialized_count, and UNINITIALIZED_PROFILE.
Referenced by verify_flow_info().
|
inline |
References initialized_p(), m_val, and max_count.
|
inlinestatic |
References from_gcov_type().
Referenced by adjust_clone_incoming_counts(), adjust_for_ipa_scaling(), autofdo::afdo_annotate_cfg(), autofdo::afdo_calculate_branch_prob(), autofdo::afdo_propagate_circuit(), autofdo::afdo_propagate_edge(), apply_probability(), apply_scale(), check_bb_profile(), profile_probability::combine_with_count(), combine_with_ipa_count(), compatible_p(), compute_alignments(), compute_branch_probabilities(), compute_function_frequency(), compute_path_counts(), consider_split(), copy_bb(), copy_cfg_body(), decide_about_value(), determine_unlikely_bbs(), drop_profile(), cgraph_node::dump(), duplicate_loop_body_to_header_edge(), eliminate_partially_redundant_load(), estimate_bb_frequencies(), execute_sm_if_changed(), expensive_function_p(), find_subloop_latch_edge_by_profile(), find_traces(), force_edge_cold(), get_info_about_necessary_edges(), gimple_find_sub_bbs(), final_warning_record::grow_type_warnings(), handle_missing_profiles(), init_caller_stats(), insert_check_and_trap(), ipa(), ipa_devirt(), ipa_inline(), ipa_merge_profiles(), ipa_propagate_frequency(), ipa_propagate_frequency_1(), ipcp_cloning_candidate_p(), isolate_path(), loop_count_in(), make_forwarder_block(), make_forwarders_with_degenerate_phis(), max(), maybe_hot_count_p(), move_sese_region_to_fn(), operator+(), operator+=(), operator-(), operator-=(), operator<(), operator<=(), operator>(), operator>=(), optimize_bb_for_size_p(), cgraph_node::optimize_for_size_p(), perhaps_add_new_callers(), possible_polymorphic_call_targets(), probability_in(), probably_never_executed(), profile_record_check_consistency(), propagate_freq(), propagate_unlikely_bbs_forward(), rebuild_frequencies(), redirect_to_unreachable(), sanitize_hot_paths(), scale_bbs_frequencies_profile_count(), scale_dominated_blocks_in_loop(), scale_strictly_dominated_blocks(), shrink_wrap_one_built_in_call_with_conds(), simd_clone_adjust(), split_function(), ssa_fix_duplicate_block_edges(), to_cgraph_frequency(), to_frequency(), to_sreal_scale(), try_shrink_wrapping(), unlikely_executed_bb_p(), unlikely_executed_edge_p(), unloop_loops(), update_bb_profile_for_threading(), update_counts_for_self_gen_clones(), update_profile(), update_profile_for_new_sub_basic_block(), update_profiling_info(), and update_specialized_profile().
|
private |
Referenced by adjusted_zero(), afdo(), apply_probability(), apply_probability(), apply_scale(), apply_scale(), dump(), force_nonzero(), from_gcov_type(), global0(), global0adjusted(), guessed(), guessed_local(), guessed_zero(), ipa(), ipa_p(), max(), merge(), ok_for_merging(), operator+(), operator+=(), operator-(), operator-=(), operator==(), precise_p(), probability_in(), quality(), reliable_p(), stream_in(), stream_out(), stream_out(), uninitialized(), and verify().
|
private |
Referenced by adjusted_zero(), apply_probability(), apply_probability(), apply_scale(), apply_scale(), differs_from_p(), dump(), force_nonzero(), from_gcov_type(), guessed_zero(), initialized_p(), max(), nonzero_p(), operator+(), operator+=(), operator-(), operator-=(), operator<(), operator<(), operator<=(), operator<=(), operator==(), operator>(), operator>(), operator>=(), operator>=(), probability_in(), stream_in(), stream_out(), stream_out(), to_cgraph_frequency(), to_gcov_type(), to_sreal_scale(), uninitialized(), value(), verify(), and very_large_p().
|
static |
|
static |
Referenced by estimate_bb_frequencies().
|
staticprivate |
Referenced by initialized_p(), uninitialized(), and verify().