We are a small team of data scientists, running our models on our local machines, data is stored in SQL Server. We are looking for some approaches which will help us with scalability,create and run scheduled jobs, monitor the scheduled runs, evaluate performance, compare with existing model and report.
It’s a big topic all together. Let me try to give me a quick overview on the tools and techniques you need to have in place for productionization, monitoring and evaluating results from ML models. I think, first you need to move away from your local machine to a centralized on-prem/cloud server with R and/or Python installed (If you use any of these). Then, on the requirement either you can create a webservice (Flask and Gunicorn for Python based application(Open source and free) / Plumber for R(has some limitations) schedule (cronjob in unix server/ any advanced ETL tool you use in your Organization. There are various other ways as well for which you might need to spend some money. You can check out some of my insights on deploying models in production in this article. Once you deploy, you need to monitor your model performance continuously and you need to check the data drift and variations in the input data for your model. If there is a drastic change on the input data for your top attributes in scoring model which differs from your training data, then it is the time to retrain your model. You can check out my article to explore some ways to monitor model performance on an ongoing basis. You might need to take snapshots of your results to compare them with the actuals when it is available to report your accuracy metrics. Hope this helps.