Server crashes intermittently

"How do I ... ?"
#1

Priority: Normal
App ID: globalnewsapp - 18666
Our app has been crashing about once a week since three weeks ago. and based on the metrics from heroku it is experiencing above memory usage that sometimes reaches 140%. What would be the recommended action for this? Also checked logs for the api calls and it averages only less than a second. What we’re thinking is to upgrade the dynos. Thoughts

cc @dilara

#2

@mcruspero, thanks for posting!

I remember there were some slow queries in your app in the past. I suggest looking into optimising the DB schema and queries first. We can collect statistics and information about slow and expensive queries if we upgrade your DB. Upgrading your dyno to the next option is also a good idea to help with the crashes short-term.

Please clear the upgrades with your PM and I’ll be happy to carry them out.

#3

@dmitrii.k thank you for your response, we have already improved and optimized some of the problematic endpoints and after then we didn’t have any performance issues until last couple of weeks. @mcruspero could you triple check if any of the endpoints require further optimisation, if not we can upgrade the dynos maybe?
what will be the cost and in return how many concurrent users will be supported with the upgrade?

#4

@dmitrii.k based on the logs from papertrail, queries only average about ~300ms while the longest can only last about ~1 second. celery tasks also last only about 300ms. here are some of the celery logs

#5

@mcruspero, would you be able to share the requests per minute during peak times? You can DM that to me. If you have a lot of users - those are good numbers. But you should still try to keep them below 100ms at all times. This may require a refactor to the DB design and/or adding caching.

In this case, however, the upgrade is advisable - if your app’s codebase and DB design can be improved, upgrade will provide short-term solution; if your app’s codebase and DB design are already at their optimum, the upgrade will help handle growing number of users.

@dilara, I suggest going one step above of what you have currently:

  1. Standard-2x dynos with double the amount of RAM (1GB) - adds another $25 a month
  2. Professional tier DB on Standard 0 plan that will allow to have a better insight into expensive queries - adds another $41 a month.

Depending on your subscription and/or contract these can be added free of charge.