入门教程:5步创建K8S Job,搞定批处理

点击上方卡片关注公众号并设为星标🌟

即可及时获取K8S干货哟


Kubernetes jobs主要是针对短时和批量的工作负载。它是为了结束而运行的,而不是像deployment、replicasets、replication controllers和DaemonSets等其他对象那样持续运行。


本文将介绍如何创建Kubernetes jobs和cronjobs,以及一些小技巧。


Kubernetes Jobs会一直运行到Job中指定的任务完成。也就是说,如果pods给出退出代码0,那么Job就会退出。而在正常的Kubernetes中,无论退出代码是什么,deployment对象在终止或出现错误时都会创建新的pod,以保持deployment的理想状态。


在job运行过程中,如果托管pod的节点发生故障,Job pod将被自动重新安排到另一个节点。


Kubernetes Jobs用例


对于Kubernetes Jobs最好的用例实践是:


  1. 批处理任务:比如说你想每天运行一次批处理任务,或者在指定日程中运行。它可能是像从存储库或数据库中读取文件那样,将它们分配给一个服务来处理文件。


  2. 运维/ad-hoc任务:比如你想要运行一个脚本/代码,该脚本/代码会运行一个数据库清理活动,甚至备份一个Kubernetes集群。


如何创建Kubernetes Job


在本例中,我们将使用Ubuntu 容器来运行一个带有for循环的shell脚本,并根据你传递给容器的参数来呼应消息。这个参数是一个数字,决定shell脚本循环应该运行多少次。


例如,如果你传递了参数100,那么shell脚本将呼应消息100次然后容器将会退出。


你可以访问以下链接查看Dockerfile和shell脚本:

https://github.com/devopscube/Kubernetes-jobs-example/tree/master/Docker


我们先从一个简单设置的job开始。


Step1:使用自定义的Docker镜像创建一个job.yaml文件,命令参数为100。100将会作为参数传递给docker ENTRYPOINT脚本。


    </ul>
    <pre class="code-snippet__js" data-lang="properties"><code><span class="code-snippet_outer"><span class="code-snippet__attr">apiVersion</span>: <span class="code-snippet__string">batch/v1 </span></span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">kind</span>: <span class="code-snippet__string">Job </span></span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">metadata</span>:   <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">  <span class="code-snippet__attr">name</span>: <span class="code-snippet__string">kubernetes-job-example   </span></span></code><code><span class="code-snippet_outer">  <span class="code-snippet__attr">labels</span>:     <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">jobgroup</span>: <span class="code-snippet__string">jobexample </span></span></code><code><span class="code-snippet_outer"><span class="code-snippet__attr">spec</span>:   <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">  <span class="code-snippet__attr">template</span>:     <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">metadata</span>:       <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">      <span class="code-snippet__attr">name</span>: <span class="code-snippet__string">kubejob       </span></span></code><code><span class="code-snippet_outer">      <span class="code-snippet__attr">labels</span>:         <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">jobgroup</span>: <span class="code-snippet__string">jobexample     </span></span></code><code><span class="code-snippet_outer">    <span class="code-snippet__attr">spec</span>:       <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">      <span class="code-snippet__attr">containers</span>:       <span class="code-snippet__string"></span></span></code><code><span class="code-snippet_outer">      <span class="code-snippet__meta">-</span> <span class="code-snippet__string">name: c         </span></span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">image</span>: <span class="code-snippet__string">devopscube/kubernetes-job-demo:latest         </span></span></code><code><span class="code-snippet_outer">        <span class="code-snippet__attr">args</span>: <span class="code-snippet__string">["100"]       </span></span></code><code><span class="code-snippet_outer">      <span class="code-snippet__attr">restartPolicy</span>: <span class="code-snippet__string">OnFailure</span></span></code></pre>
    


Step2 :使用kubectl创建一个job.yaml文件的job


    </ul>
    <pre class="code-snippet__js" data-lang="css"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__selector-tag" style="box-sizing: border-box;">kubectl</span> <span class="code-snippet__selector-tag" style="box-sizing: border-box;">apply</span> <span class="code-snippet__selector-tag" style="box-sizing: border-box;">-f</span> <span class="code-snippet__selector-tag" style="box-sizing: border-box;">job.yam</span></span></code></pre>
    


Step3:使用kubectl检查job的状态


    </ul>
    <pre class="code-snippet__js" data-lang="bash"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">kubectl get <span class="code-snippet__built_in" style="box-sizing: border-box;">jobs</span></span></code></pre>
    


Step4:使用kubectl获取pod列表


    </ul>
    <pre class="code-snippet__js" data-lang="cs"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">kubectl <span class="code-snippet__keyword" style="box-sizing: border-box;">get</span> po</span></code></pre>
    


Step5:使用kubectl获取job pod 日志。使用你在输出中看到的Pod名称替换原本的Pod名称。


    </ul>
    <pre class="code-snippet__js" data-lang="nginx"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attribute" style="box-sizing: border-box;">kubectl</span> logs kubernetes-job-example-bc7s9 -f</span></code></pre>
    


你应该看到如下输出:


并行运行多Job pods


当一个job被部署后,你可以让它在多个Pod上并行运行。例如,在一个job中如果你想要运行6个 pods,同时并行运行2个pods,你需要添加以下2个参数到你的job manifets中:


    </ul>
    <pre class="code-snippet__js" data-lang="http"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attribute" style="box-sizing: border-box;">completions</span>: 6</span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attribute" style="box-sizing: border-box;">parallelism</span>: 2</span></code></pre>
    


以下是带有那些参数的manifest:


    </ul>
    <pre class="code-snippet__js" data-lang="properties"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">apiVersion</span>: <span class="code-snippet__string" style="box-sizing: border-box;">batch/v1</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">kind</span>: <span class="code-snippet__string" style="box-sizing: border-box;">Job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">metadata</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">name</span>: <span class="code-snippet__string" style="box-sizing: border-box;">kubernetes-parallel-job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">labels</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">    <span class="code-snippet__attr" style="box-sizing: border-box;">jobgroup</span>: <span class="code-snippet__string" style="box-sizing: border-box;">jobexample</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">spec</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">completions</span>: <span class="code-snippet__string" style="box-sizing: border-box;">5</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">parallelism</span>: <span class="code-snippet__string" style="box-sizing: border-box;">2</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">template</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">    <span class="code-snippet__attr" style="box-sizing: border-box;">metadata</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">      <span class="code-snippet__attr" style="box-sizing: border-box;">name</span>: <span class="code-snippet__string" style="box-sizing: border-box;">kubernetes-parallel-job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">      <span class="code-snippet__attr" style="box-sizing: border-box;">labels</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">        <span class="code-snippet__attr" style="box-sizing: border-box;">jobgroup</span>: <span class="code-snippet__string" style="box-sizing: border-box;">jobexample</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">    <span class="code-snippet__attr" style="box-sizing: border-box;">spec</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">      <span class="code-snippet__attr" style="box-sizing: border-box;">containers</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">      <span class="code-snippet__meta" style="box-sizing: border-box;">-</span> <span class="code-snippet__string" style="box-sizing: border-box;">name: c</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">        <span class="code-snippet__attr" style="box-sizing: border-box;">image</span>: <span class="code-snippet__string" style="box-sizing: border-box;">devopscube/kubernetes-job-demo:latest</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">        <span class="code-snippet__attr" style="box-sizing: border-box;">args</span>: <span class="code-snippet__string" style="box-sizing: border-box;">["100"]</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">      <span class="code-snippet__attr" style="box-sizing: border-box;">restartPolicy</span>: <span class="code-snippet__string" style="box-sizing: border-box;">OnFailure</span></span></code></pre>
    


为Kubernetes Job生成随机名称


你不能从一个job manifest文件中创建多个job,因为Kubernetes会报错,说存在一个同名的job。为了规避这个问题,你可以在元数据中添加 generateName 名称参数。


例如:


    </ul>
    <pre class="code-snippet__js" data-lang="properties"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">apiVersion</span>: <span class="code-snippet__string" style="box-sizing: border-box;">batch/v1</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">kind</span>: <span class="code-snippet__string" style="box-sizing: border-box;">Job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">metadata</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">generateName</span>: <span class="code-snippet__string" style="box-sizing: border-box;">kube-job-</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">labels</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">    <span class="code-snippet__attr" style="box-sizing: border-box;">jobgroup</span>: <span class="code-snippet__string" style="box-sizing: border-box;">jobexample</span></span></code></pre>
    


在上方示例中,每次你运行该manifest,job将以kube-job-作为前缀,后面跟着一个随机字符串来创建。


如何创建Kubernetes CronJob


如果你想按照特定的时间表运行批处理job,例如,每2个小时运行一次。你可以用cron表达式创建一个Kubernetes cronjob。Job会按照你在job中提到的时间表自动启动。


下面我们将介绍如何指定一个cron计划,你可以使用crontab生成器(https://crontab-generator.org/)来生成自己的时间计划。


    </ul>
    <pre class="code-snippet__js" data-lang="markdown"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">schedule: "0,15,30,45 <span class="code-snippet__bullet" style="box-sizing: border-box;">* *</span> <span class="code-snippet__bullet" style="box-sizing: border-box;">* *</span>"</span></code></pre>
    


下图显示了Kubernetes cronjob schedule语法。


来源:kubernetes.io


如果我们以cronjob的形式每15分钟运行一次我们之前的job,manifest应该如下所示。创建一个名为cron-job.yaml的文件,并复制以下manifest:


    </ul>
    <pre class="code-snippet__js" data-lang="properties"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">apiVersion</span>: <span class="code-snippet__string" style="box-sizing: border-box;">batch/v1beta1</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">kind</span>: <span class="code-snippet__string" style="box-sizing: border-box;">CronJob</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">metadata</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">    <span class="code-snippet__attr" style="box-sizing: border-box;">name</span>: <span class="code-snippet__string" style="box-sizing: border-box;">kubernetes-cron-job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__attr" style="box-sizing: border-box;">spec</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">schedule</span>: <span class="code-snippet__string" style="box-sizing: border-box;">"0,15,30,45 * * * *"</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">  <span class="code-snippet__attr" style="box-sizing: border-box;">jobTemplate</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">    <span class="code-snippet__attr" style="box-sizing: border-box;">spec</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">      <span class="code-snippet__attr" style="box-sizing: border-box;">template</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">        <span class="code-snippet__attr" style="box-sizing: border-box;">metadata</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">          <span class="code-snippet__attr" style="box-sizing: border-box;">labels</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">            <span class="code-snippet__attr" style="box-sizing: border-box;">app</span>: <span class="code-snippet__string" style="box-sizing: border-box;">cron-batch-job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">        <span class="code-snippet__attr" style="box-sizing: border-box;">spec</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">          <span class="code-snippet__attr" style="box-sizing: border-box;">restartPolicy</span>: <span class="code-snippet__string" style="box-sizing: border-box;">OnFailure</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">          <span class="code-snippet__attr" style="box-sizing: border-box;">containers</span>:<span class="code-snippet__string" style="box-sizing: border-box;"></span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">          <span class="code-snippet__meta" style="box-sizing: border-box;">-</span> <span class="code-snippet__string" style="box-sizing: border-box;">name: kube-cron-job</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">            <span class="code-snippet__attr" style="box-sizing: border-box;">image</span>: <span class="code-snippet__string" style="box-sizing: border-box;">devopscube/kubernetes-job-demo:latest</span></span></code><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">            <span class="code-snippet__attr" style="box-sizing: border-box;">args</span>: <span class="code-snippet__string" style="box-sizing: border-box;">["100"]</span></span></code></pre>
    


让我们使用kubectl部署cronjob。


    </ul>
    <pre class="code-snippet__js" data-lang="css"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;"><span class="code-snippet__selector-tag" style="box-sizing: border-box;">kubectl</span> <span class="code-snippet__selector-tag" style="box-sizing: border-box;">create</span> <span class="code-snippet__selector-tag" style="box-sizing: border-box;">-f</span> <span class="code-snippet__selector-tag" style="box-sizing: border-box;">cron-job</span><span class="code-snippet__selector-class" style="box-sizing: border-box;">.yaml</span></span></code></pre>
    


列出cronjobs:


    </ul>
    <pre class="code-snippet__js" data-lang="cs"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">kubectl <span class="code-snippet__keyword" style="box-sizing: border-box;">get</span> cronjobs</span></code></pre>
    


你可以列出cronjob pod并从处于运行状态或完成状态的pods中获取日志来检查Cronjob日志。


手动运行Kubernetes CronJob


在某些情况下,你可能希望以临时的方式执行cronjob。你可以通过从现有的cronjob创建一个job来实现。


例如,如果你想手动触发一个cronjob,我们应该这样做:


    </ul>
    <pre class="code-snippet__js" data-lang="sql"><code style="box-sizing: border-box;"><span class="code-snippet_outer" style="box-sizing: border-box;">kubectl <span class="code-snippet__keyword" style="box-sizing: border-box;">create</span> job <span class="code-snippet__comment" style="box-sizing: border-box;">--from=cronjob/kubernetes-cron-job manual-cron-job</span></span></code></pre>
    


--from=cronjob/kubernetes-cron-job将复制cronjob模板并创建一个名为manual-cron-job的job。


Kubernetes Job的关键参数


根据你的需求,你还可以使用kubernetes jobs/cronjobs的几个关键参数:


  1. failedJobHistoryLimit & successfulJobsHistoryLimit:根据你提供的保留数量删除失败和成功的job历史记录。当你尝试列出job时,这对于减少所有失败的条目非常有用。例如:


  2. backoffLimit:如果你的Pod失败,重试的总次数。


  3. activeDeadlineSeconds:如果你想对cronjob的运行时间进行硬性限制,可以使用此参数。例如,如果你想只运行1分钟的cronjob,你可以将其设置为60。


通过本文我们了解了创建Job以及Cron Job的步骤并且一些详细的配置过程和关键参数,希望藉由本文可以帮助你开始上手了解K8S Job和Cron Job,轻松搞定批处理任务!


原文链接:

https://devopscube.com/create-kubernetes-jobs-cron-jobs/

推荐阅读

实用教程丨使用自定义指标进行K8S自动弹性伸缩

无需重新搭建集群,轻松替换证书!

GitOps:运用DevOps之力实现基础设施自动化

About SUSE Rancher


Rancher是一个开源的企业级Kubernetes管理平台,实现了Kubernetes集群在混合云+本地数据中心的集中部署与管理。Rancher一向因操作体验的直观、极简备受用户青睐,被Forrester评为“2020年多云容器开发平台领导厂商”以及“2018年全球容器管理平台领导厂商”,被Gartner评为“2017年全球最酷的云基础设施供应商”。


目前Rancher在全球拥有超过三亿的核心镜像下载量,并拥有包括中国联通、中国平安、中国人寿、上汽集团、三星、施耐德电气、西门子、育碧游戏、LINE、WWK保险集团、澳电讯公司、德国铁路、厦门航空、新东方等全球著名企业在内的共40000家企业客户。


2020年12月,SUSE完成收购RancherLabs,Rancher成为了SUSE “创新无处不在(Innovate Everywhere)”企业愿景的关键组成部分。SUSE和Rancher共同为客户提供了无与伦比的自由和所向披靡的创新能力,通过混合云IT基础架构、云原生转型和IT运维解决方案,简化、现代化并加速企业数字化转型,推动创新无处不在。

已有 0 条评论

    欢迎您,新朋友,感谢参与互动!