Broken GRUB2, rescue mode.

[ifoolb]

Hi people,
Now every time I boot my laptop grub runs into this:
http://192.227.167.69/IMAG0057.jpg
Say, there is an error message which I've googled around but found nothing related to it. To enter the normal grub boot menu, I have to manually insmod linux16, then invoke initrd16, though followed by another error message there, which seems harmless.

This is my disk info:

[ifoolb@y400 ~]$ lsblk
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0   477G  0 disk 
└─sda1               8:1    0    56G  0 part 
  ├─ssd_lvm2-usr00 253:0    0    40G  0 lvm  /usr
  ├─ssd_lvm2-root  253:1    0    10G  0 lvm  /
  └─ssd_lvm2-swap  253:2    0     6G  0 lvm  [SWAP]
sdb                  8:16   0 931.5G  0 disk 
├─sdb1               8:17   0     2M  0 part 
├─sdb2               8:18   0   720G  0 part 
│ ├─lvm2-home      253:3    0   680G  0 lvm  /home
│ ├─lvm2-var       253:4    0    30G  0 lvm  /var
│ └─lvm2-tmp       253:5    0    10G  0 lvm  /tmp
└─sdb3               8:19   0    10G  0 part /boot
sr0                 11:0    1  1024M  0 rom

As grub can find /boot on startup, I don't know if above is useful though, I have two disks in my laptop, a hdd(sdb) and a ssd (sda). My experience with the installation is hard. I encountered various problems since the ssd had been in. In short, the laptop boots without problem when fedora is first installed, till I install grub-customizer to modify grub2 theme and screen resolution, and then reboot, grub menu show beautifully, but at this point all kernels (maybe) become unbootable. I see only a black screen with a blink cursor at the top left corner of the screen after I select a kernel, and it doesn't respond to any key press, including ALT+CTRL+DEL. I then boot with a liveusb, chroot into the system, install grub with

grub2-install --target=i386-pc /dev/sdb

and then

grub2-mkconfig -o /boot/grub2/grub.cfg

everything works normally without errors. But it falls into rescue shell after reboot just like what's in the image. I tried adding

GRUB_PRELOAD_MODULES=linux16

in /etc/default/grub and/or

insmod linux16

into /boot/grub2/grub.cfg, but none worked. I then tried compiling grub2 from source, but the problem persists. Thanks for reading so far, can someone help with this?

[dobbi]

Longshot.

- Maybe you just edited in the wrong place instead of GRUB_PRELOAD_MODULES you should have used the GRUB_CMDLINE_LINUX.

GRUB2 caveats in Linux.

4.3.2 GNU/Linux

It is relatively easy to boot GNU/Linux from GRUB, because it somewhat resembles to boot a Multiboot-compliant OS.

    Set GRUB’s root device to the same drive as GNU/Linux’s. The command search --set=root --file /vmlinuz or similar may help you (see search).
    Load the kernel using the command linux (see linux):

    grub> linux /vmlinuz root=/dev/sda1

    If you need to specify some kernel parameters, just append them to the command. For example, to set acpi to ‘off’, do this:

    grub> linux /vmlinuz root=/dev/sda1 acpi=off

    See the documentation in the Linux source tree for complete information on the available options.

    With linux GRUB uses 32-bit protocol. Some BIOS services like APM or EDD aren’t available with this protocol. In this case you need to use linux16

    grub> linux16 /vmlinuz root=/dev/sda1 acpi=off

    If you use an initrd, execute the command initrd (see initrd) after linux:

    grub> initrd /initrd

    If you used linux16 you need to use initrd16:

    grub> initrd16 /initrd

    Finally, run the command boot (see boot). 

Caution: If you use an initrd and specify the ‘mem=’ option to the kernel to let it use less than actual memory size, you will also have to specify the same memory size to GRUB. To let GRUB know the size, run the command uppermem before loading the kernel. See uppermem, for more information.

source:
https://www.gnu.org/software/grub/ma...#GNU_002fLinux

After reading that.

# cat /etc/default/grub
GRUB_TIMEOUT="10"
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT="saved"
GRUB_DISABLE_SUBMENU="true"
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="linux16 rd.lvm.lv=fedora/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
export GRUB_COLOR_NORMAL="light-gray/black"
export GRUB_COLOR_HIGHLIGHT="yellow/green"

- Edit it as highlighted in the GRUB_CMDLINE_LINUX part.

# grub2-mkconfig -o /boot/grub2/grub.cfg

If that doesn't work I would be looking at something that was left behind by something else.
That pointer error looks to me like a pointer was created and then left behind after GRUB was updated and so it needs to be erased, but I don't know where those things are installed exactly I would have to look it up which is another problem since I don't know where at the moment.

ps: probably the MBR portion, I thought that number could be a header magic number but it is too short is 8 bits long not 16 so it can't be that.
https://www.gnu.org/software/grub/ma...ison-technique

[ifoolb]

# cat /etc/default/grub
GRUB_TIMEOUT="10"
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT="saved"
GRUB_DISABLE_SUBMENU="true"
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="linux16 rd.lvm.lv=fedora/swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=fedora/root $([ -x /usr/sbin/rhcrashkernel-param ] && /usr/sbin/rhcrashkernel-param || :) rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
export GRUB_COLOR_NORMAL="light-gray/black"
export GRUB_COLOR_HIGHLIGHT="yellow/green"

- Edit it as highlighted in the GRUB_CMDLINE_LINUX part.

# grub2-mkconfig -o /boot/grub2/grub.cfg

If that doesn't work I would be looking at something that was left behind by something else.
That pointer error looks to me like a pointer was created and then left behind after GRUB was updated and so it needs to be erased, but I don't know where those things are installed exactly I would have to look it up which is another problem since I don't know where at the moment.

ps: probably the MBR portion, I thought that number could be a header magic number but it is too short is 8 bits long not 16 so it can't be that.
https://www.gnu.org/software/grub/ma...ison-technique

Unfortunately, that doesn't work either. I think GRUB2 is independent from the kernel to be booted, since all modules reside in /boot/grub2/i386-pc ( in my case, for legecy, efi off), if I understand correctly. From a ubuntu tutorial page of grub, it says gurb will first try to enter 'normal' mode if things are alright, or it will fall into a rescue shell. In my case, when it's in the rescue shell, if I issue 'normal' command, it says 'Unknow command normal', obviously it's not loaded. But if I invoke insmod normal, that mysterious error

error: symbol `grub_file_progress_hook' not found

will apper again, until I insert lnux16 and enter the normal boot menu by employing the method mentioned above, many modules are unable to load with the same error message. As for the pointer error, I don't know what a pointer is in grub stuff. I'll dive into it later. The first error gets priority, however, from the google results, it seems that this goes into source code level stuff, I'll try to understand these things before it takes me too much time.

[dobbi]

That error is probably caused by the missed pointer there, it should be pointing to somewhere where that info resided then since it doesn't find it and can't load the "symbols" that I think refer to the ELF symbol tables.

https://blogs.oracle.com/ali/entry/i..._symbol_tables

That is just conjecture I don't really know, my best guess is that the MBR was modified and it is pointing to the wrong position and the only way I know how to check that is to copy the MBR and annotate its location than do the changes and do a diff to see what changed and try to figure it why it changed, again this is just a guess.

[ifoolb]

That error is probably caused by the missed pointer there, it should be pointing to somewhere where that info resided then since it doesn't find it and can't load the "symbols" that I think refer to the ELF symbol tables.

https://blogs.oracle.com/ali/entry/i..._symbol_tables

That is just conjecture I don't really know, my best guess is that the MBR was modified and it is pointing to the wrong position and the only way I know how to check that is to copy the MBR and annotate its location than do the changes and do a diff to see what changed and try to figure it why it changed, again this is just a guess.

I'm a little getting lost in combination of these things: pointer, symbols, ELF, MBR. This seems beyond the scrope of simple grub configurations, but comes to compilation, linking. As far as I know, symbol looking up happens when a program is trying to call some function in separate libraries, ELF is the binary execuatable format used by Linux. Since My knowledge is weak at how Linux boots, I don't know the releationship between grub binary and MBR... I'll read the post the link refers to, and learn something about MBR, because I think I'm using GPT partition schme, however, with which I'm not very familar either. So thanks for your pointer, I now at least know which direction to go further. And I remember you, you helped me with some other problems, you're an excellent guy! I'll return to the thread when I finished some reading.

[dobbi]

Well I said that because of this.

/* a.out symbol table
valid only if (flags & 0x0010)		(MB_INFO_AOUT_SYMS)*/
	    aout_symbol_table_t aout_sym;

source: http://geezer.osdevbrasil.net/osd/boot/grub-how.txt
Why does it support the table system? there is a reason for it.

But seeing the flags there I suspect the symbol means the internal flags.
It needs it for some reason.

UPDATE: Wed 22 Oct 07:03:20 EDT 2014

Another clue about what those symbols may mean.

$ egrep -ri aout_sym $HOME/Downloads/grub/grub-2.00
$HOME/Downloads/grub/grub-2.00/ChangeLog:	* include/multiboot.h (struct multiboot_aout_symbol_table)
$HOME/Downloads/grub/grub-2.00/ChangeLog:	(multiboot_aout_symbol_table_t, multiboot_elf_section_header_table_t)
$HOME/Downloads/grub/grub-2.00/include/multiboot.h:#define MULTIBOOT_INFO_AOUT_SYMS		0x00000010
$HOME/Downloads/grub/grub-2.00/include/multiboot.h:struct multiboot_aout_symbol_table
$HOME/Downloads/grub/grub-2.00/include/multiboot.h:typedef struct multiboot_aout_symbol_table multiboot_aout_symbol_table_t;
$HOME/Downloads/grub/grub-2.00/include/multiboot.h:    multiboot_aout_symbol_table_t aout_sym;

Still there are things to explore.

There is a program to check the scripts for syntax errors.

[root@tentacles ~]# grub2-script-check --usage
Usage: grub2-script-check [-v?V] [--verbose] [--help] [--usage] [--version]
            [PATH]

There is other stuff to, you can get a complete list of available utilities for grub issuing "info grub" on the command line.

The source code for it I downloaded from here.

Source code for GRUB-2 (28/06/2012 09:00:00)
ftp://ftp.gnu.org/gnu/grub/grub-2.00.tar.xz
ftp://ftp.gnu.org/gnu/grub/grub-2.00.tar.xz.sig

I tried greping it but no love.

[nsquid@tentacles ~]$ egrep -ri e6ee $HOME/Downloads/grub/grub-2.00/
$HOME/Downloads/grub/grub-2.00/grub-core/lib/libgcrypt/cipher/tiger.c:  U64_C(0x2860971a6b943fcd) /*   60 */, U64_C(0x3dde6ee212e30446) /*   61 */,
$HOME/Downloads/grub/grub-2.00/grub-core/lib/libgcrypt-grub/cipher/tiger.c:  U64_C(0x2860971a6b943fcd) /*   60 */, U64_C(0x3dde6ee212e30446) /*   61 */,

I also did search for 0xe6ee and the only reference I found was about Unicode, which I can't discard as possibility because of this.
grub-2.00/unicode/UnicodeData.txt

The MBR has "pointer" to call the rest of the bootloader if it is pointing to the wrong place it would break things how it will break I can't remember, it has been a long time since I last had boot problems of this sort, so I don't know if it would just stop working or if it would allow GRUB to issue commands.

https://en.wikipedia.org/wiki/Master_boot_record

Update: Wed 22 Oct 06:50:58 EDT 2014

To copy the mbr.

dd if=/dev/sda of=$HOME/Downloads/mbr/mbrsda.bak bs=512 count=1

source:
http://www.cyberciti.biz/faq/howto-copy-mbr/
https://wiki.archlinux.org/index.php/Master_Boot_Record

To see if anything changed:

diff file1 file2

source: https://stackoverflow.com/questions/...g-binary-files

To see what changed.

xxd f1 > f1.hex
xxd f2 > f2.hex

then

diff f1.hex f2.hex

or

yum install vbindiff

[ifoolb]

Thanks for the exploration and explanation, I've read some wiki about GPT and MBR. And I also spent much time researching its source codes, I'm very sleepy now, but there are things I want to list here:
1, Those 'XXX_hook' stuff are something look like call back functions, which are very likely used by some modules, or grub itself. I've looked through all source codes but haven't found that 'grub_file_progress_hook'.

2, I'm too careless, actually my manually compiled grub is located in /usr/local/sbin rather than /usr/sbin.

3, I tried modifying the source codes to prevent looking for 'grub_file_progress_hook', but others were broken, different error shown.

4, I can't simply use my manually compiled grub-install to overwrite the existing grub2, which leads to a chain loading. (The original grub prior to my grub).

I'll read your post and continue exploring later...

UPDATE Wed Oct 22 23:07:40 EDT 2014:

Whatever I can't find that hook function anywhere, including disassembly of grub2 that comes from Fedora. I guess it might be some compilation error of grub2, because other symbols are missing in the copy of grub2 that I compiled. I dumped MBR from /dev/sdb (my boot disk), but I don't know which file to 'diff' with, in my case, it's different from /boot/grub2/i386-pc/boot.img.

I made a mistake, I issued the command cat /boot/grub2/i386-pc/boot.img > /dev/sdb even if I'm not sure whether boot.img shound be put into MBR, but anyway, I've lost all my data...

So now it will take me more time to reinstall system, however, I think it is reproduceable. I'll redo the process, and make further decision accordingly...

[ifoolb]

I finally managed to boot the system. What I've done is simply switching to UEFI, though I don't know why it works and the other doesn't, I can describe all cases which works and which doesn't. At first, my partition type is GPT, with single disk, it boots in legacy mode, as described in Wikipedia, Linux and BSD are support GPT booting in legacy mode. But when the second disk is inserted, that becomes another story, first of all a liveusb can't boot any more, blink cursur, not responding to keyboard, black screen. Only liveusb created by live-cd-iso-to-disk can boot, but with problem, I have to enter linux0 to boot it(thanks to it, this gives me a chance to repair). And system installed with this liveusb, even if it's somewhat usable, it stops booting when the kernel, accurately , grub is updated, just like the situation a normal liveusb will be in when startup. So I'd like to say it's anaconda that does the magical trick to let it boot, and also, in this situation /boot can't be on my ssd, it has to be on HDD, or it says can't find /dev/sdX where /boot resides during startup. While in UEFI mode everything that's a problem formerly does not trouble at all, wherever /boot resides is ok, if only not in unreasonable places like a volume group. Therefore I can assume that the problem is not necessarily caused by GRUB and it may well be from within the hardware settings, at least to say GRUB is not capable of doing something like GPT in legacy mode with two disks in my laptop(Lenovo Y400). Therefore I might have gone into wrong place to look for an answer in the source codes and instead, I should mostly read documentations of both UEFI and GPT. I'll return to the thread and put some conclusions after I complete learning what these things do exactly on booting.