最近在工作中发现了一个问题，这个问题就是MySQL的磁盘满了，将数据库目录data移动到/data3目录，/etc/my.cnf里面也修改了相应的datadir目录，权限也赋予了，但是service mysql start;的时候报错，下面话不多说了，来一起看看详细的解决方法吧。
error log显示如下：
2017-09-15 16:01:01 2420 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead.
2017-09-15 16:01:01 2420 [Note] Plugin 'FEDERATED' is disabled.
^G/usr/sbin/mysqld: Can't find file: './mysql/plugin.frm' (errno: 13 - Permission denied)
2017-09-15 16:01:01 2420 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2017-09-15 16:01:01 2420 [Note] InnoDB: Using atomics to ref count buffer pool pages
2017-09-15 16:01:01 2420 [Note] InnoDB: The InnoDB memory heap is disabled
2017-09-15 16:01:01 2420 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-09-15 16:01:01 2420 [Note] InnoDB: Memory barrier is not used
2017-09-15 16:01:01 2420 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-09-15 16:01:01 2420 [Note] InnoDB: Using Linux native AIO
2017-09-15 16:01:01 2420 [Note] InnoDB: Using CPU crc32 instructions
2017-09-15 16:01:01 2420 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-09-15 16:01:02 2420 [Note] InnoDB: Completed initialization of buffer pool
2017-09-15 16:01:02 2420 [ERROR] InnoDB: ./ibdata1 can't be opened in read-write mode
2017-09-15 16:01:02 2420 [ERROR] InnoDB: The system tablespace must be writable!
2017-09-15 16:01:02 2420 [ERROR] Plugin 'InnoDB' init function returned error.
2017-09-15 16:01:02 2420 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-09-15 16:01:02 2420 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-09-15 16:01:02 2420 [ERROR] Aborting
去先检查plugin.frm权限，frm是MySQL表结构定义文件，通常frm文件是不会损坏的，但是如果出现特殊情况出现frm文件损坏也不要放弃希望，当修复MyISAM和InnoDB表时，MySQL服务会首先去调用frm文件，所以我们只能通过修复frm文件进行后面的数据恢复。
然后我们发现plugin.frm有权限，也是mysql属主：
root@hutaojie-1-pdd-sh:/data1/mysql/mysql# ll plugin.*
-rwxrwxrwx 1 mysql mysql 8586 Mar 6 2016 plugin.frm*
-rwxrwx--x 1 mysql mysql 116 Mar 6 2016 plugin.MYD*
-rwxrwx--x 1 mysql mysql 2048 Mar 6 2016 plugin.MYI*
root@hutaojie-1-pdd-sh:/data1/mysql/mysql#
google之后发现，原来问题在os这里，ubatu的mysql通过yum安装或者rpm安装的时候，会建一个/etc/apparmor.d/usr.sbin.mysqld 文件，如果数据目录不在这里面，则会报错，
/usr/sbin/mysqld: Can't find file: ‘./mysql/plugin.frm' (errno: 13 - Permission denied)
所以解决办法是在里面加上新的datadir目录。
root@huayuan:/var/lib# vim /etc/apparmor.d/usr.sbin.mysqld
# vim:syntax=apparmor
# Last Modified: Tue Jun 19 17:37:30 2007
#include <tunables/global>
/usr/sbin/mysqld {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/user-tmp>
#include <abstractions/mysql>
#include <abstractions/winbind>
capability dac_override,
capability sys_resource,
capability setgid,
capability setuid,
network tcp,
/run/mysqld/mysqld.pid rw,
/run/mysqld/mysqld.sock w,
/sys/devices/system/cpu/ r,
#.........这里面写新的datadir目录，写2行，一行r，一行rwk。
/data3/mysql/ r,
/data3/mysql/** rwk,
# Site-specific additions and overrides. See local/README for details.
#include <local/usr.sbin.mysqld>
}
然后重启mysql实例，ok，问题解决。