Forum | Documentation | Website | Blog

Skip to content
Snippets Groups Projects
  • Darrick J. Wong's avatar
    xfs: verify buffer, inode, and dquot items every tx commit · 150bb10a
    Darrick J. Wong authored
    
    generic/388 has an annoying tendency to fail like this during log
    recovery:
    
    XFS (sda4): Unmounting Filesystem 435fe39b-82b6-46ef-be56-819499585130
    XFS (sda4): Mounting V5 Filesystem 435fe39b-82b6-46ef-be56-819499585130
    XFS (sda4): Starting recovery (logdev: internal)
    00000000: 49 4e 81 b6 03 02 00 00 00 00 00 07 00 00 00 07  IN..............
    00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 10  ................
    00000020: 35 9a 8b c1 3e 6e 81 00 35 9a 8b c1 3f dc b7 00  5...>n..5...?...
    00000030: 35 9a 8b c1 3f dc b7 00 00 00 00 00 00 3c 86 4f  5...?........<.O
    00000040: 00 00 00 00 00 00 02 f3 00 00 00 00 00 00 00 00  ................
    00000050: 00 00 1f 01 00 00 00 00 00 00 00 02 b2 74 c9 0b  .............t..
    00000060: ff ff ff ff d7 45 73 10 00 00 00 00 00 00 00 2d  .....Es........-
    00000070: 00 00 07 92 00 01 fe 30 00 00 00 00 00 00 00 1a  .......0........
    00000080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
    00000090: 35 9a 8b c1 3b 55 0c 00 00 00 00 00 04 27 b2 d1  5...;U.......'..
    000000a0: 43 5f e3 9b 82 b6 46 ef be 56 81 94 99 58 51 30  C_....F..V...XQ0
    XFS (sda4): Internal error Bad dinode after recovery at line 539 of file fs/xfs/xfs_inode_item_recover.c.  Caller xlog_recover_items_pass2+0x4e/0xc0 [xfs]
    CPU: 0 PID: 2189311 Comm: mount Not tainted 6.9.0-rc4-djwx #rc4
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20171121_152543-x86-ol7-builder-01.us.oracle.com-4.el7.1 04/01/2014
    Call Trace:
     <TASK>
     dump_stack_lvl+0x4f/0x60
     xfs_corruption_error+0x90/0xa0
     xlog_recover_inode_commit_pass2+0x5f1/0xb00
     xlog_recover_items_pass2+0x4e/0xc0
     xlog_recover_commit_trans+0x2db/0x350
     xlog_recovery_process_trans+0xab/0xe0
     xlog_recover_process_data+0xa7/0x130
     xlog_do_recovery_pass+0x398/0x840
     xlog_do_log_recovery+0x62/0xc0
     xlog_do_recover+0x34/0x1d0
     xlog_recover+0xe9/0x1a0
     xfs_log_mount+0xff/0x260
     xfs_mountfs+0x5d9/0xb60
     xfs_fs_fill_super+0x76b/0xa30
     get_tree_bdev+0x124/0x1d0
     vfs_get_tree+0x17/0xa0
     path_mount+0x72b/0xa90
     __x64_sys_mount+0x112/0x150
     do_syscall_64+0x49/0x100
     entry_SYSCALL_64_after_hwframe+0x4b/0x53
     </TASK>
    XFS (sda4): Corruption detected. Unmount and run xfs_repair
    XFS (sda4): Metadata corruption detected at xfs_dinode_verify.part.0+0x739/0x920 [xfs], inode 0x427b2d1
    XFS (sda4): Filesystem has been shut down due to log error (0x2).
    XFS (sda4): Please unmount the filesystem and rectify the problem(s).
    XFS (sda4): log mount/recovery failed: error -117
    XFS (sda4): log mount failed
    
    This inode log item recovery failing the dinode verifier after
    replaying the contents of the inode log item into the ondisk inode.
    Looking back into what the kernel was doing at the time of the fs
    shutdown, a thread was in the middle of running a series of
    transactions, each of which committed changes to the inode.
    
    At some point in the middle of that chain, an invalid (at least
    according to the verifier) change was committed.  Had the filesystem not
    shut down in the middle of the chain, a subsequent transaction would
    have corrected the invalid state and nobody would have noticed.  But
    that's not what happened here.  Instead, the invalid inode state was
    committed to the ondisk log, so log recovery tripped over it.
    
    The actual defect here was an overzealous inode verifier, which was
    fixed in a separate patch.  This patch adds some transaction precommit
    functions for CONFIG_XFS_DEBUG=y mode so that we can detect these kinds
    of transient errors at transaction commit time, where it's much easier
    to find the root cause.
    
    Signed-off-by: default avatarDarrick J. Wong <djwong@kernel.org>
    Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
    150bb10a