Apache YuniKorn
 | |
| Developer(s) | Apache Software Foundation |
|---|---|
| Repository | Yunikorn Repository |
| Written in | Go |
| Engine | |
| Operating system | Cross-platform |
| License | Apache License 2.0 |
| Website | yunikorn |
Search Apache YuniKorn on Amazon.
Apache YuniKorn is an open-source, lightweight resource scheduler that supports large-scale, multi-tenant, and cloud-native environments. Apache YuniKorn also adapts to different ResourceManager YuniKorn's architecture design implementations like Apache Hadoop YARN [1] [2] Its primary purpose was to replace Apache YARN. As for scheduling Apache Spark on Kubernetes, it has now become the widely accepted method. [3]
On a large scale, in multi-tenant settings, and in dynamically created cloud-native environments, YuniKorn facilitates efficient fine-grained resource sharing for different Spark workloads. By providing a solid foundation for a range of applications, including large-scale data transformation, analytics, and machine learning, YuniKorn enables Apache Spark to become an enterprise-grade fundamental platform for users.[4]
History
In order to build a contemporary method for scheduling massive, distributed operations in a future where clouds predominate, Apache YuniKorn was initially created at Cloudera in 2019.[5] The small group of individuals were searching for a truly solid solution to the scheduling and Kubernetes scheduling issue that they had seen. At that time, they also believed that Yarn and Kubernetes should be merged, therefore they decided on a single scheduling method. Apache incubation began on January 21, 2020, and the stable version 0.8.0 was published on May 4, 2020, nearly six months later.[6]
Naming
The meaning of Yunikorn [Y YARN + K Kubernetes + Uni Unified][7] and the purpose of a scheduler is to support any container orchestration system both on-cloud and on-premise use cases.[8]
Development
Apache YuniKorn is developed under the Apache License 2.0[9] by the Apache YuniKorn Community within the Apache Software Foundation. The project is driven by over 25 Apache committers[10]
Overview
Apache YuniKorn is a resource scheduler that is designed for scheduling and managing larger dynamic applications on the container orchestration structure and became an Apache Top Level Project in March 2022.[11] [12]
The use of conventional scheduling techniques with big data tools has a number of shortcomings. Due to the absence of fine-grained control over autoscaling in the majority of resource schedulers, resource usage is often out of sync, autoscaling periods for both upscaling and downscaling are prolonged, and as a result, cloud costs are higher and throughput/performance are decreased.
Resource consumption is enhanced and autoscaling performance is boosted thanks to YuniKorn's Gang scheduling and bin-packing.[13] YuniKorn simultaneously ran periodic Apache Spark tasks and saw almost a 2-fold increase in throughput, with a 2-fold decrease in average job runtime and a 3-fold decrease in scale-up and scale-down latencies for 200 nodes. [14]
There are four components in YuniKorn:[15]
- yunikorn-core
- yunikorn-web
- yunikorn-scheduler-interface
- yunikorn-k8shim
Releases
| Version | Original release date | Latest version[16] | Release date[17] | |
|---|---|---|---|---|
| 0.8.0 | 2020-05-04 | 0.8.0 | 2020-05-04 | |
| 0.9.0 | 2020-08-28 | 0.9.0 | 2020-08-28 | |
| 0.10.0 | 2021-04-09 | 0.10.0 | 2021-04-09 | |
| 0.11.0 | 2021-08-18 | 0.11.0 | 2021-08-18 | |
| 0.12.1 | 2021-12-26 | 0.12.1 | 2021-12-26 | |
| 0.12.2 | 2022-02-03 | 0.12.2 | 2022-02-03 | |
| 1.0.0 | 2022-05-06 | 1.0.0 | 2022-05-06 | |
| 1.1.0 | 2022-09-08 | 1.1.0 | 2022-09-08 | |
Legend: Old version Older version, still supported Latest version Latest preview version | ||||
References
- ↑ "Welcome to Apache YuniKorn | Apache YuniKorn". yunikorn.apache.org.
- ↑ "Apache YuniKorn - A Universal Scheduler". The Apache Software Foundation. 24 October 2022.
- ↑ Desk, AIT News (2022-05-17). "The Apache Software Foundation Announces Apache YuniKorn as a Top-Level Project". AiThority. Retrieved 2022-11-22.
- ↑ "Apache Spark on Kubernetes: How Apache YuniKorn helps - openthings". my.oschina.net. Retrieved 2022-11-22.
- ↑ "Apache YuniKorn becomes a Top-Level Project". DataXP. 2022-05-16. Retrieved 2022-11-22.
- ↑ "The Apache Software Foundation Announcement". sg.news.yahoo.com. Retrieved 2022-11-22.
- ↑ Yang, WeiWei (17 July 2019). "YuniKorn: a universal resources scheduler". Cloudera Blog.
- ↑ Schmidt, Julia (2019-07-18). "YuniKorn to unify cross-platform scheduling • DEVCLASS". DEVCLASS. Retrieved 2022-11-22.
- ↑ Apache YuniKorn License, The Apache Software Foundation, 2022-11-22, retrieved 2022-11-22
- ↑ "Contributors to apache/yunikorn-core". GitHub. Retrieved 2022-11-22.
- ↑ Foundation, The Apache Software (2022-05-16). "The Apache Software Foundation Announces Apache® YuniKorn™ as a Top-Level Project". GlobeNewswire News Room. Retrieved 2022-11-22.
- ↑ "The Apache Software Foundation Announces Apache® YuniKorn™ as a Top-Level Project". news.apache.org. 16 May 2022.
- ↑ Mayer, Marton (2021-09-02). "Optimizing Cloudera Data Engineering Autoscaling Performance". Cloudera Blog. Retrieved 2022-11-22.
- ↑ datafoam (2021-09-03). "Optimizing YuniKorn Autoscaling Performance". Datafoam. Retrieved 2022-11-22.
- ↑ "Apache YuniKorn Repro". The Apache Software Foundation. 23 August 2022.
- ↑ "yunikorn-release". yunikorn-release. Retrieved 2022-11-22.
- ↑ "Apache YuniKorn Releases". yunikorn.apache.org. Retrieved 2022-11-22.
External links
This article "Apache YuniKorn" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:Apache YuniKorn. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
