Argo Workflow

Argo Workflows is the most popular workflow execution engine for Kubernetes. Light-weight, scalable, and easier to use. Designed from the ground up for containers without the overhead and limitations of legacy VM and server-based environments. Cloud agnostic and can run on any Kubernetes cluster.

Its Kinda like Gitea or Github Actions for Kubernetes

why argo

• reference earlier work
  • prophet forecast via flask arm image
  • unblogged telegraf darts plugin that tried to do all in one via telegraf
• messy with nodejs cron and raspberrypi os cron
• goal
  • automated ETL, model training and model inference pipeline
    • extract
    • transform / train / predict
    • load
  • robust scheduled workflows
• requirements
  • run on arm
  • clean separated ETL steps
  • control over compute scalability of steps
  • possibility to scale heavy steps to gpu / cloud
• tried
  • for work, a host of tools (base10, mlops, deeploy)
  • telegraf (unfriendly plugin structure)
  • kubeflow (doesnt run on arm)
  • mlflow (no schedulable workflows)
  • argo
• answer: argo

key points

• arm64 support out of the box
• language agnostic with declarative yaml files
• utilize containers natively

from resource: Argo Workflows MLOps meetup

pro’s

• do one job, do it well
• built on kubernetes

why argo for ml

• scheduled workflows (finally a product that does this well)
• store artifacts and metrics
• argo manages execution of containers

workflow

• define steps, each step has
  • image
  • command that runs in container
• steps can fork and merge
• defined in workflow.yaml file
  • metadata
  • specs
    • templates (reusable units of execution)
    • entrypoint (template the workflow will start with)

scheduling

• cron like annotations to easily run workflows on time schedule

failure handling

• specify retry strategy
  • retry conditions
  • limit number of retries
• specify whether steps can run concurrently (and possibly result in duplicated data)

artifacts

• possibility to forward artifects to next steps
• easily debug steps because you can see the input and output of steps
• cloud agnostic

Argo vs Kubeflow

Argo’s primary focus is on workflow management, while Kubeflow is a platform for ML development that uses Argo to create its workflows on …

I’m an ml engineer and had to make the same choice half a year ago. In the end I chose for argo. The top reason for doing so is because it works equally well with all machine learning frameworks. Kubeflow focuses too much (IMHO) on tensorflow. —pietermarsman reddit

references

• https://argoproj.github.io/
• usecases for machine learning
  • Automating Research Workflows at BlackRock
  • Bridging into Python Ecosystem with Cloud-Native Distributed Machine Learning Pipelines
  • Building Medical Grade AI with Argo Workflows
  • CI/CD for Machine Learning at MLB using Argo Workflows - Eric Meadows
  • Dynamic, Event-Driven Machine Learning Pipelines with Argo Workflows
  • Machine Learning as Code: GitOps for ML with Kubeflow and Argo CD
  • Machine Learning with Argo and Ploomber
  • Making Complex R Forecast Applications Into Production Using Argo Workflows
  • MLOps at TripAdvisor: ML Models CI/CD Automation with Argo - Ang Gao (Principal Software Engineer, TripAdvisor)
  • Towards Cloud-Native Distributed Machine Learning Pipelines at Scale