Model Evalutation Dashboard
Tyra is a prediction model evaluation dashboard used internally in DSaPP for several projects. In DSaPP, our general pipeline will produce millions of models from either open source machine learning libraries or our own algorithms following the scikit-learn interface. Millions of models are generated by the grid search of all combinations of hyper parameters, different feature sets, different lables and different training and predicting time windows. Due to different goals and interventions of each project, we don’t just care about a single metric (accuracy, precision or whatever metrics), but have to take lots of other factors into considertation such as feture distribution, response distribution, stability overtime, individual feature importance, global feature importance, jaccard similarity of the result and etc. The evaluation process becomes a multi-dimentional problem. Tyra is created for this purpose to look deeper into the model, using postgreSQL, React.js, d3.js and Flask.
