Investing the Effects of Overcommitting YARN resources

Investing the Effects of Overcommitting YARN resources

• 1.
  Investigating the Effectsof Overcommitting YARN Resources Jason Lowe [email protected]
• 2.
  Problem: Underutilized ClusterResources
• 3.
  Optimize The Jobs! ●Internal Downsizer tool quantifies job waste ● Application framework limitations ● Optimally tuned container can still have opportunities Time ContainerUtilization Underutilized Resources
• 4.
  What about StaticOvercommit? ● Configure YARN to use more memory than node provides ● Tried with some success ● Performs very poorly when node fully utilized
• 5.
  Overcommit Prototype DesignGoals ● No changes to applications ● Minimize changes to YARN protocols ● Minimize changes to scheduler internals ● Overcommit on memory only ● Conservative growth ● Rapid correction
• 6.
  Overcommit Overview ResourceManager NodeManager Utilizationreport in heartbeat ■ Unaware of overcommit amount ■ Self-preservation preemption ■ Adjusts internal node size ■ Assigns containers based on new size Application Masters
• 7.
  NodeMemoryNodeUtilization ResourceManager Node Scaling Time Time NoOvercommit Reduced Overcommit Full Overcommit Allocated Node Mem Total Node Mem Original Node Mem
• 8.
  ResourceManager Overcommit Tunables ParameterDescription Value memory.max-factor Maximum amount a node will be overcommitted 1.5 memory.low-water-mark Maximum overcommit below this node utilization 0.6 memory.high-water-mark No overcommit above this node utilization 0.8 memory.increment-mb Maximum increment above node allocation 16384 increment-period-ms Delay between overcommit increments if node container state does not change 0 Parameters use yarn.resourcemanager.scheduler.overcommit. prefix
• 9.
  NodeManager Self-Preservation Preemption NodeUtilization High Water Mark Low Water Mark ● Utilization above high mark triggers preemption ● Preempts enough to reach low mark utilization ● Does not preempt containers below original node size ● Containers preempted in group order ○ Tasks from preemptable queue ○ ApplicationMasters from preemptable queue ○ Tasks from non-preemptable queue ○ ApplicationMasters from non-preemptable queue ● Youngest containers preempted first within a group 0% 100%
• 10.
  NodeManager Overcommit Tunables ParameterDescription Value memory.high-water-mark Preemption when above this utilization 0.95 memory.low-water-mark Target utilization after preemption 0.92 Parameters use yarn.nodemanager.resource-monitor.overcommit. prefix
• 11.
  Results
• 12.
  Results - Capacity_Gainedvs Work_Lost
• 13.
  Lessons Learned ● Significantovercommit achievable on real workloads ● Far less preemption than expected ● Container reservations can drive overcommit growth ● Coordinated reducers can be a problem ● Cluster totals over time can be a bit confusing at first
• 14.
  Future Work ● YARN-5202 ●Only grows cluster as a whole not individual queues ● Nodes can overcommit while others are relatively idle ● CPU overcommit ● Predict growth based on past behavior ● Relinquish nodes during quiet periods ● Integration with YARN-1011
• 15.
  YARN-1011 ● Explicit GUARANTEEDvs. OPPORTUNISTIC distinction ● Promotion of containers once resources are available ● SLA guarantees along with best-effort load
• 16.
  Acknowledgements ● Nathan Robertsfor co-developing overcommit POC ● Inigo Goiri for nodemanager utilization collection and reporting ● Giovanni Matteo Fumarola for nodemanager AM container detection ● YARN-1011 contributors for helping to shape the long-term solution
• 17.
  Questions? Jason Lowe [email protected]