赞同 4
分享
刷新

Celery导致MySQL崩溃

2020-06-14 Bug Man 4 55
简介：造成这个问题原因我也没有找到，主要是为了分享一种解决问题的心态。天下所有的学问都是记问之学，不会的东西先记下来遇见高人了再适当的去寻求一下答案，当然自己解决问题的能力是最重要也是最关键的。
  2020.06.14
  Bug Man
  4
  55
  172.17.0.1
  中国.上海
 
 

先描述一下我遇到的问题：

最近我想要把celery部署到线上，之前都是命令行里面直接跑起来的，现在需要把定时任务放到线上。然后一切都已经做完了，我一启动一个定时任务后台直接访问不到MySQL。我但是就觉得这事不妙，然后开始查看MySQL服务的状态果然挂了。

我第一反应不是看日志，而是将这些服务再重启一次。然后结果还是一样只要你勾选定时任务的enable一保存MySQL 准挂掉~。查看了MySQL服务状态之后，我就开始有点懵了这个以前也没见过。

MySQL日志信息
2020-06-14T00:03:56.896141Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-06-14T00:03:56.897697Z 0 [Note] /xxx/xxx/ysqld (mysqld 5.7.29) starting as process 24879 ...
2020-06-14T00:03:56.900317Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-14T00:03:56.900342Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-14T00:03:56.900347Z 0 [Note] InnoDB: Uses event mutexes
2020-06-14T00:03:56.900350Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-14T00:03:56.900354Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-14T00:03:56.900357Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-14T00:03:56.900594Z 0 [Note] InnoDB: Number of pools: 1
2020-06-14T00:03:56.900691Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-14T00:03:56.902010Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-06-14T00:03:56.902056Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2020-06-14T00:03:56.902063Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2020-06-14T00:03:56.902068Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-06-14T00:03:56.902075Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-06-14T00:03:56.902078Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-06-14T00:03:56.902082Z 0 [ERROR] Failed to initialize builtin plugins.
2020-06-14T00:03:56.902085Z 0 [ERROR] Aborting

2020-06-14T00:03:56.902102Z 0 [Note] Binlog end
2020-06-14T00:03:56.902146Z 0 [Note] Shutting down plugin 'CSV'
2020-06-14T00:03:56.902326Z 0 [Note] /xxx/xxx/mysqld: Shutdown complete

我搜了一下第一行的这个警告，发现很多的人文章说要再MySQL配置文件中添加一行explicit_defaults_for_timestamp=true。这个MySQL配置文件，就根据你放置的位置有关了，我就不展示我的操作了，systemctl resatrt mysqld.service重启mysql服务。 然后再次启动celery服务，然后MySQL直接崩，为什么呢？之前崩溃的时候数据库里的定时任务状态还是enable，所以再服务都起来之后继续崩溃。

MySQL新的报错日志
2020-06-14T00:10:52.788707Z 0 [Note] /xxx/xxx/mysqld (mysqld 5.7.29) starting as process 25419 ...
2020-06-14T00:10:53.072885Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-06-14T00:10:53.072931Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-06-14T00:10:53.072936Z 0 [Note] InnoDB: Uses event mutexes
2020-06-14T00:10:53.072940Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-06-14T00:10:53.072946Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-06-14T00:10:53.072950Z 0 [Note] InnoDB: Using Linux native AIO
2020-06-14T00:10:53.073284Z 0 [Note] InnoDB: Number of pools: 1
2020-06-14T00:10:53.073405Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-06-14T00:10:53.169002Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-06-14T00:10:53.169064Z 0 [ERROR] InnoDB: mmap(137428992 bytes) failed; errno 12
2020-06-14T00:10:53.169076Z 0 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2020-06-14T00:10:53.169085Z 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2020-06-14T00:10:53.169096Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2020-06-14T00:10:53.169102Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2020-06-14T00:10:53.169117Z 0 [ERROR] Failed to initialize builtin plugins.
2020-06-14T00:10:53.169121Z 0 [ERROR] Aborting

2020-06-14T00:10:53.169147Z 0 [Note] Binlog end
2020-06-14T00:10:53.174010Z 0 [Note] Shutting down plugin 'CSV'
2020-06-14T00:10:53.174376Z 0 [Note] /xxx/xxx/mysqld: Shutdown complete

然后我通过Cannot allocate memory for the buffer pool这条信息，我以为是内存不够用了，然后我top看了一下好像也没有啊。管不了那么多了，先重启下先解决问题后找原因，果然重启之后celery和mysql不冲突了。这个原因还需要再去查一查，可能是因为内存不够用了也可能是某一时刻服务请求量太大了，这个我需要找个高人问问了，自己太菜了。

这个问题是我前一天晚上到了12点了还没解决，然后这个时候不能因为这个问题熬夜去找，因为半夜人的心可能会安静一点但是对于解决问题帮助也不会有很大的。我觉得解决问题方向是最重要的，如果方向不对那就是南辕北辙精力再多也不够使，其次就是各个方面知识的积累最有助于你快速解决问题了，再次就是耐心反复查看报错信息是什么意思。