Find slow data processing tasks (before your customers do)
Here are some of the ways you can discover your data processing jobs are too slow:
- Jobs start getting killed when they hit timeouts.
- Customers start complaining about slow or failed jobs.
- Your cloud computing bill is twice what it was last month.
While these notification mechanisms do work, it’s probably best not to rely on them.
Life is easier when jobs finish successfully, customers are happy, and you have plenty of money left over in your budget.
That means you want to identify unexpected slowness or high memory usage before the situation get that bad.
The sooner you can identify performance problems, the sooner you can fix them.
So how can you identify inefficient tasks in your data pipeline or workflow?
Let’s find out!
From “always