Somewhat unfortunate name clash with the now-back-burnered deep learning framework written in Clojure, also called Cortex: https://github.com/originrose/cortex
LOL...every other AI related project these days is called some variation or part of:
Cortex
Mental
Mind
Neural/Neurons
Which is, of course, hilarious, since none of them have anything to do with the brain or mind whatsoever, being slightly advanced computational statistics programs.
Pretty cool name (cortex), but also taken by this project - https://github.com/cortexproject/cortex - "A multitenant, horizontally scalable Prometheus as a Service"
I work on ML infrastructure at some big company, so I'm always interested to see whats new in this field. This seems to be a thin wrapper around tf.estimator, which provides a majority of the functionality. The only novel things are yaml config for defining some basic transformations and data format. It doesn't seem super useful, am I missing something?
The main thing we try to help with is orchestrating Spark, TensorFlow, TensorFlow Serving, and other workloads without requiring you to manage the infrastructure. You’re right that we have a thin layer around tf.estimator (by design) because our goal is to make it easy to create scalable and reproducible pipelines from building blocks that people are familiar with. We translate the YAML blocks into workloads that run as a DAG on Kubernetes behind the scenes.
We use TesnorFlow serving (https://www.tensorflow.org/serving) for serving the trained models. We also run Flask to transform the incoming JSON to match the way the data has been transformed at training time.
We have a lot of respect for the work that the KubeFlow team is doing. Their focus seems to be on helping you deploy a wide variety of open source ML tooling to Kubernetes. We use a more narrow stack and focus more on automating common workflows.
For example, we take a fully declarative approach; the “cortex deploy” command is a request to “make it so”, rather than “run this training job”. Cortex determines at runtime exactly what pipeline needs to be created to achieve the desired state, caching as aggressively as it can (e.g. if a hyperparameter to one model changes, only that model is re-trained and re-deployed, whereas if a transformer is updated, all transformed_columns which use that transformer are regenerated, all models which use those columns are re-trained, etc). We view it as an always-on ML application, rather than a one-off ML workload.
Which is, of course, hilarious, since none of them have anything to do with the brain or mind whatsoever, being slightly advanced computational statistics programs.
Really, if you want people to use your thing, make it easy for them to find.
For example, we take a fully declarative approach; the “cortex deploy” command is a request to “make it so”, rather than “run this training job”. Cortex determines at runtime exactly what pipeline needs to be created to achieve the desired state, caching as aggressively as it can (e.g. if a hyperparameter to one model changes, only that model is re-trained and re-deployed, whereas if a transformer is updated, all transformed_columns which use that transformer are regenerated, all models which use those columns are re-trained, etc). We view it as an always-on ML application, rather than a one-off ML workload.