Mysterious Grubby Beasts...and How to Fix Them...
FedoraForum.org - Fedora Support Forums and Community
Results 1 to 7 of 7
  1. #1
    Join Date
    May 2019
    Location
    California
    Posts
    26
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Mysterious Grubby Beasts...and How to Fix Them...

    As you probably know, the 5.1.1-300.fc30 kernel update pushed out a couple of days ago, and I did my upgrade willingly and without fear of reprisal from my inanimate companions. However, several days before the upgrade appeared, I realized that my rescue kernel was very old and I wanted to rebuild it. So, in my ill-found hubris, I tried to run the spiffy remove/rebuild commands:

    #rm -f /boot/vmlinuz-0-rescue-* /boot/initramfs-0-rescue-*.img (I ran this after realizing that I couldn't just rebuild with an existing rescue kernel in place)

    #/etc/kernel/postinst.d/51-dracut-rescue-postinst.sh $(uname -r) /boot/vmlinuz-$(uname -r) -- This script should have a warning if an existing kernel exists and a switch option to force the overwrite.

    Apparently, my linuxy self-delusion included the assumption that the released kernel tools would be able to cleanly rebuild the rescue kernel based on the installed kernel version, since it is a P1, mission critical function and nobody would ever release updates without ensuring that the system could create the necessary rescue tools. Right? Right, Fedora?

    Unfortunately (or maybe business-as-usual in the world of linux), the rebuild command generated errors complaining about missing new-kernel-pkg, similar to previously-reported bugs from over a year ago, as I later discovered. Anyway, I foolishly tried to extract new-kernel-pkg from the deprecated grubby, which probably left me with a screwed up grubby install. And...the command still failed after that. But, still, I remained undaunted. Everything was still working OK.

    Until...I ran the 5.1.11 update. The kernel update scriptlets generated errors at the very end. I looked at the yum logs, but didn't see the scriptlet failures there. Thankfully, at least, it looks like removing the old rescue kernel forced the rescue rebuild...but then the grub build derailed. Not realizing the significance of the scriptlet failure, I then rebooted into a failed/no-boot scenario with the following message on-screen:

    Code:
    error: ../../grub-core/script/lexer.c:337:syntax error.
    error: ../../grub-core/script/lexer.c:337:Incorrect command.
    error: ../../grub-core/script/lexer.c:337:syntax error.
    fatal error: token too large, exceeds YYLMAX
    Arghhh...so, I booted into rescue mode to see if I could fix it. When I immediately checked /etc/grub2-efi.cfg (alias to /boot/efi/EFI/fedora/grub.cfg), I saw that it was huge (over 600k) and filled with garbage. So, I removed it, re-ran grub2-mkconfig, and everything was fine again. Relatively easy-to-fix. Maybe the swearing helped.

    Forensically speaking, one failing scriptlet was obviously the one that generated the corrupted grub.cfg and another was the one that set the new kernel as the default boot. So, I simply ran grub2-set-default on the new kernel name to fix it.

    Attached is the corrupted grub.cfg file in case anyone wants to look at it with a binary editor. The top of the file appears normal and the end is filled with garbage lvm name recursion, binary spew and trailing script text.

    I have posted the above as both a cautionary tale and a how-to to fix your boot if something like this happens to you. A little bit of knowledge is a dangerous thing sometimes. Also, the most important observation is that if you want to generate a new rescue kernel, the regular install will probably do it. All you have to do is delete the existing one first right before a new kernel update occurs. However, in my case, by default, the update does not overwrite the existing one (based on the fact that mine was very old), and I didn't realize it would be fixed at the next kernel update if I deleted it.

    My question is:
    I have removed grubby-deprecated and made sure I deleted new-kernel-pkg. What do I need to do to prevent this problem from happening in the future and/or to restore things the way they should be? Thanks!
    Attached Files Attached Files

  2. #2
    Join Date
    May 2019
    Location
    California
    Posts
    26
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Mysterious Grubby Beasts...and How to Fix Them...

    Well...it doesn't look like anyone had any clues for me on this one. I tried to revert back to the recommended installation and in the latest kernel updates grub.cfg has been OK. However, grubenv has still been screwed up after each update, so I have had to manually update it. I noticed today that there was a new grubby release - v8.40.31.fc30.x86_64. So, I am hoping that they fixed the grubenv update problem that occurs during the post-install scriptlet. It has already been reported as a bug.

    The error is always the same, as follows:
    Code:
    grub2-editenv: error: environment block too small.
    ...and then it never updates grubenv.

    Also, I have been deleting the rescue kernel to force a rebuild for each kernel update, but I'm not sure that's working either. The last one I tried didn't work (timed out trying to load LVMs), but I haven't tried the new one for 5.1.17-300 yet. Maybe it will be better.

    Anyway...so the grub.cfg corruption noted above appears to be gone, but the grubenv and rescue kernel building are still not right, I think.
    Last edited by RocketHat; 19th July 2019 at 08:38 PM.

  3. #3
    Join Date
    May 2019
    Location
    California
    Posts
    26
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Mysterious Grubby Beasts...and How to Fix Them...

    I tested 5.1.18-300 today, and the scriptlet is still generating the same error and the rescue kernel doesn't work properly. So, I guess I will have to report these issues.

  4. #4
    Join Date
    May 2019
    Location
    California
    Posts
    26
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Mysterious Grubby Beasts...and How to Fix Them...

    I found the bug for grub2-editenv here: https://bugzilla.redhat.com/show_bug.cgi?id=1625124

    I guess I didn't really understand the fixed nature of the grubenv file, but now I do. Anyway, I was able to fix the "environment block too small" problem by recreating the grubenv file with the following commands to ensure that it was exactly 1024 bytes, etc., as suggested in Post #23 of the above bug:

    Code:
    n.b. The commands below assume you are using grub2 under EFI.
    cd /boot/efi/EFI/fedora  (change to the grub directory on FC30)
    rm grubenv (get rid of the old, non-1024 byte grubenv file that is causing the "too small" error because it is...wait for it...too small! :D)
    grub2-editenv grubenv create (create a new grubenv of the correct size with no eols, etc.)
    grub2-editenv grubenv set saved-entry=vmlinuz-5.1.18-300.fc30.x86.64 (set the default boot selection -- whatever that may be on your system)
    grub2-editenv grubenv [other environment config commands you want to execute] (add other grubenv commands, as desired)

    No more environment block too small error. But honestly, Fedora, the kernel update post-install scriptlet should automatically fix any issues with grubenv, especially stripping out eols, entering the current kernel version as a saved entry and fixing the padding to ensure the file is exactly 1024 bytes instead of just failing.

    So...now I'm down to just the non-working rescue kernel issue. And the moral of the story is, use grub2-editenv that will automatically adjust the padding, assuming it is correct to begin with (even though it should auto-correct it) and avoid editing grubenv manually...or if you do, make sure you do it in binary noeol mode and that it is exactly 1024 bytes.
    Last edited by RocketHat; 23rd July 2019 at 11:20 PM.

  5. #5
    Join Date
    May 2019
    Location
    California
    Posts
    26
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Mysterious Grubby Beasts...and How to Fix Them...

    I finally had some time to do some testing after the latest kernel update and filed the rescue bug here:
    Fedora Bug #1739796.

    So...maybe it will lead to some resolution of my final kernel booting issue.

  6. #6
    Join Date
    Aug 2009
    Posts
    364
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Mysterious Grubby Beasts...and How to Fix Them...

    I never use those grub scripts. I edit my grub.cfg myself. Keep a copy so when grub decides to update, I just copy it back. Mine fully readable, whereas original grub.cfg is full of nonsense(obviously, my opinion). All those variables and such. Been doing it this way since original grub, years ago.

  7. #7
    Join Date
    May 2019
    Location
    California
    Posts
    26
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Re: Mysterious Grubby Beasts...and How to Fix Them...

    I understand. I'm OK with the auto-generated version, but, like you, also back up the working config/env before upgrades. I haven't had time to dig into grub syntax yet. But...as long as the boot list is there and everything works....

    In this case, the final kernel issue for me is the rescue kernel boot failure. I'm hoping that there will be some positive resolution from those who are more linuxy than I am and that it may also fix the potential bug for others.

Similar Threads

  1. grubby not useful with Fedora
    By danespen in forum Using Fedora
    Replies: 4
    Last Post: 28th October 2018, 10:54 PM
  2. [SOLVED]
    Grubby problem in F16
    By sidebrnz in forum Using Fedora
    Replies: 18
    Last Post: 7th December 2011, 10:27 PM
  3. Die grubby
    By QoolB in forum Using Fedora
    Replies: 2
    Last Post: 14th July 2005, 03:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •