Hey,
I am big supporter of canva and congrats on this and other products. Could you explain why did you not end up using tools like ray hyper parameters and many others like w and b etc
There are many open and commercial options for hyperparameter optimization.
We didn't select Argo for hyperparameter optimization. The situation is that because we already used Argo for our ML training and data processing jobs, we wanted to see if it could be extended for hyperparameter optimization. It actually can!
As mentioned briefly in the article, we try to use tools with minimal overlap. It's not always possible of course and sometimes new tools are necessary.
Just to add: the MLOps ecosystem evolves very quickly.
Tools that were once favored before can become obsolete very quickly. The general philosophy is to keep a constrained and minimal toolset: if we have a general purpose tool that could be extended to other cases too, then we do that. Moreover any introduced tool has tradeoffs not always apparent: additional staff training and maintainence costs.
That being said, it'd be interesting to see how the space evolves over the next few years. I think we'll eventually see best practices arising from the space, but for now it's very nascent.
I was going to ask the same thing. Why would a company the size of Canva recreate the wheel rather than leveraging tools such as SigOpt that have already applied their expertise in handling the batch hyperparameter suggestions?
On a positive note, the orchestration piece does seem pretty slick, I suppose offering more than Ray alone.
Yes, YAML does seem to be the "Cloud Native" tax. For all but the simplest of Workflows, it is necessary to use a templating language or one of the Argo SDKs.
We use jsonnet for building and manipulating YAML at the moment and are reasonably happy with it.
There are many open and commercial options for hyperparameter optimization.
We didn't select Argo for hyperparameter optimization. The situation is that because we already used Argo for our ML training and data processing jobs, we wanted to see if it could be extended for hyperparameter optimization. It actually can!
As mentioned briefly in the article, we try to use tools with minimal overlap. It's not always possible of course and sometimes new tools are necessary.
Just to add: the MLOps ecosystem evolves very quickly.
Tools that were once favored before can become obsolete very quickly. The general philosophy is to keep a constrained and minimal toolset: if we have a general purpose tool that could be extended to other cases too, then we do that. Moreover any introduced tool has tradeoffs not always apparent: additional staff training and maintainence costs.
That being said, it'd be interesting to see how the space evolves over the next few years. I think we'll eventually see best practices arising from the space, but for now it's very nascent.
On a positive note, the orchestration piece does seem pretty slick, I suppose offering more than Ray alone.
I do find Argo's turing-complete language implemented in YAML painful, so I am also curious.
Yes, YAML does seem to be the "Cloud Native" tax. For all but the simplest of Workflows, it is necessary to use a templating language or one of the Argo SDKs.
We use jsonnet for building and manipulating YAML at the moment and are reasonably happy with it.