CCTC201才云科技鄧德源-谷歌容器集群管理系統(tǒng)實踐

1、CCTC2016才云科技鄧德源-谷歌容器集群管理系統(tǒng)實谷歌容器集群管理系統(tǒng)實踐鄧德源才云科技Engineering in Google? Development? <20% time side project? work on any project you want? dashboard for posting jobs? you can do more than you think!? Be SRE for <2 or 3 months (<20% more salary!)? SRE (Site Reliability Engineering)? 50% time fo

2、r development? to automate routine tasks? scale nicely? SRE consists of two categories: 55% SDE, and 45% Ops? Say no to development team? error quota, e.g. 99.95% slo monthly means 4hrs impact? both SRE and Development team mange the risk? Ability to change code base? Postmortem? Not to blame, but t

3、o find problems and fix itContainer? Reason? primary goal: save money - VM is heavy? high-density and performance? fast to start? Start Container Journey? <2004 - ? Everything runs in container? use container for decade? &gt;<2B container a weekLimited Isolationchrootcgroupslxclmctfy<20

4、04<2006<2008<2013user namespace <2014<2007 namespaces?<2015 OCIContainerImage from RedhatLimited Isolationchrootcgroupslxclmctfy<2004<2006<2008<2013user namespace <2014<2007 namespaces?<2015 OCI? cgroup? resource isolation (cpu, memory, blockio, etc.)ContainerL

5、imited Isolationchrootcgroupslxclmctfy<2004<2006<2008<2013 user namespace <2014<2007 namespaces?<2015 OCI? cgroup v1 -&gt; v<2? multiple hierarchy? subcontainer and asymmetric isolation? v<2: unified hierarchyImage from Redhat sub1 sub<2sub3subcontainerCPU MemNetcon

6、tainer1container<2container3asymmetric isolationcg1cg10 cg11cg100 cg101ContainerLimited Isolationchrootcgroupslxclmctfy<2004<2006<2008<2013user namespace <2014<2015 OCI<2007 namespaces?Image from Toptal? Namespacesenvironment? isolate pro cess's 6><#00aa00'>v

7、iew of the operating? Mount, UTS, IPC, PID, Network, User? Support added fair recently? Primarily Linux cgroups? 'namespace' is done at user-space policiesContainer? lxc? lmctfy? internal version of Imctfy exists long before lxc? lxc: no strong abstraction? namespace abstraction + raw cgroup

8、? lxc: no programmable API? must be built to work with other tools? lmctfy: more abstraction and enhancement? e.g. subcontainer and asymmetric isolation? e.g. quality of service? OCI? Container: runtime + image? Based on DockerLimited Isolationchroot cgroupslxclmctfy<2004<2006<2008<2013u

9、ser namespace <2014<2007 namespaces?<2015 OCIContainer? Container management? clustering is the hard part? hundreds of engineerings building cluster management systemBorg: concepts? Configuration? Borg configuration language? Most hated ' language? Job and Task? Job: unit of deployment,

10、 including resources requirement, number of tasks, etc? Task: unit of running entity? Resource container: the container to run taskJobTask1 Task<2Borg: concepts? AllocSet and Alloc? co-scheduling: e.g. logsaver and application? share resources for all tasks? persist data when tasks exitJobTask1 T

11、ask<2host bind mount/cg11/cg1<2Borg: concepts? Scheduling? priority? resources: resource has priority? quota: quota has priority? packages: vs docker image? machine constraints? %ports% (BNS, chubby)? options:? repeated failures? quorum requirement? etc? Runtime? application class? resource es

12、timation:? decrease resource reservation (gradual decay function)borg config file borgcfg1RPC/Protocol BufferPrimary BorgMasterschedulerservertask1pkg1<2Paxospkg<2resource containerBorglettask_Nresource containerBorgletpollinginstalling packageBorg Cell8 'run ' + args + resourceAuthN/Z

13、Borg: Request LifecyclelinkshardBorg: The Good, The Bad, The Ugly? 高穩(wěn)定性、高自動化、高智能? 極其復(fù)雜的配置文件最流行語言排行榜？Go, Python, Java, C+, Javascript,最令人畏懼語言排行榜？Python, Borgcfg, Borgmon, Shell,? Borgmaster becomes borg monster? People step on each other 's footOmega: 早年寄予厚望? From Monolithic to Shared State? Pers

14、istent Storage as the ground truth? Look to the Future!? Collaboration across the globe? A bottom up driven projectOmega: 中年四面楚歌? Engineering mayhem? Testing cell? API rewrites, all Borg callers have to change (along with behavioral change)!? Borgmaster is hard to rewrite?向現(xiàn)實妥協(xié)：? 項目進度? 與 Borg 的關(guān)系Ome

15、ga: 晚年戛然而止? 工程與理論的差距? 預(yù)測的困難? 精確度 vs 吞吐量? Borg 的進化? 性能? 軟件工程? 終究下架? Big shuffleCloud: The Urs'quake? 經(jīng)濟上的巨大回報? 技術(shù)上的巨大領(lǐng)先? 產(chǎn)品化上的差異? PaaS與laaS的矛盾? GAE vs GCE? Managed VMKubernetes? 此處為何地? Google Mountain <#00aa00'>View? Google Pittsburgh? kubernetes 乳名 ? Kubernetes birth placeKubernetesyam

16、l config file kubectl1HTTPSingle Masterscheduler API server con1<2etcd4KubeletKubernetes Cluster3 AuthN/Zcontrollermanagercon<2infrainstalling packagewatchpostcon1Kubeletcon<2infra? 一個 Cluster 里有一個或多個 Borg cell? Borg 容器之外的其他任務(wù)? Borg 自身誰來管理？? 機器層面誰來管理？? 網(wǎng)絡(luò)層面誰來管理？? 安全誰來控制？? 鏡像如何管理？Cluster man

17、agementBorgHarewareOps流程工具/、自動系統(tǒng)安裝工具在線硬盤修復(fù)系統(tǒng)生產(chǎn)系統(tǒng)維護管理系統(tǒng)節(jié)點醫(yī)生網(wǎng)絡(luò)醫(yī)生耗竭系集群工作流管理系統(tǒng) 日歷系統(tǒng)Capacity 安全把控系統(tǒng)分布式存儲集群數(shù)據(jù)庫用戶和組管理系統(tǒng)SDN網(wǎng) 絡(luò)自動節(jié)能系統(tǒng)負(fù)載均衡資源分配擴容系統(tǒng)集群監(jiān)測系統(tǒng)安全掃描系統(tǒng)分布式構(gòu)建系統(tǒng)分布式測試系統(tǒng)調(diào)試系統(tǒng)和工具調(diào)試系統(tǒng)和工具發(fā)布管理工具The entire familyAnd . The Robot? Disk Erase? Decommission? Inventory ManagementAn example workflow: Pre-borg 集群工作流管理系統(tǒng)自動系統(tǒng)安裝工具集群數(shù)據(jù)庫節(jié)點管理agentBorgdriverChubbydriver節(jié)點醫(yī)生