Yum and RPM database corruption

[sevensevenseven]

Hello everyone,

Yum is really slow after a new install of F17 on a Lenovo W520. Have tried several times to rebuld the RPM database and cleaning yum cache but yum is just VERY slow.

I'd do the following:

sudo yum clean all
sudo rm -f /var/lib/rpm/__db.00*
sudo rpm -vv --rebuilddb
sudo yum clean all

IOTOP shows yum using up a lot of IO on the disk as if it's not happy witht he DB.

1520 be/4 root 0.00 B/s 5.10 M/s 0.00 % 97.83 % python /bin/yum update

Yum will take 10+ minutes to run transaction checks. An RPM query is very fast which makes it seem as if the DB is fine.

time rpm -qa | grep kernel
kernel-PAEdebug-devel-3.4.3-1.fc17.i686
kernel-devel-3.4.3-1.fc17.i686
kernel-PAE-3.4.4-5.fc17.i686
kernel-PAE-modules-extra-3.4.3-1.fc17.i686
kernel-PAEdebug-modules-extra-3.4.4-5.fc17.i686
kernel-PAEdebug-modules-extra-3.4.3-1.fc17.i686
kernel-devel-3.4.4-5.fc17.i686
kernel-3.4.4-5.fc17.i686
kernel-PAEdebug-3.4.4-5.fc17.i686
kernel-PAE-modules-extra-3.4.4-5.fc17.i686
kernel-PAE-3.4.3-1.fc17.i686
kernel-PAEdebug-devel-3.4.4-5.fc17.i686
kernel-3.4.3-1.fc17.i686
kernel-3.3.4-5.fc17.i686
kernel-headers-3.4.4-5.fc17.i686
kernel-PAE-devel-3.4.3-1.fc17.i686
abrt-addon-kerneloops-2.0.10-4.fc17.i686
kernel-PAE-devel-3.4.4-5.fc17.i686
libreport-plugin-kerneloops-2.0.10-3.fc17.i686
kernel-PAEdebug-3.4.3-1.fc17.i686

real 0m1.302s
user 0m1.258s
sys 0m0.060s

For example, I'd run sudo yum update and after 4 minutes, I'd still be sitting at:

time sudo yum update
Loaded plugins: langpacks, presto, refresh-packagekit
^C^C^C^C

^C^C^C^C^C^C^C
^Z
[1]+ Stopped sudo yum update

real 3m41.734s
user 0m0.000s
sys 0m0.001s

If i try to clean up yum, I get:

sudo yum clean all
error: rpmdb: BDB0113 Thread/process 1789/3077883584 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 - (-30973)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

What else could I be missing?

(Long time lurker but first time posting. Thanks to everyone for their help throughout the years. I've been running Fedora on my main laptop for several years now and have been a very happy user 99% of the time and really angry 1% of the time.)

Cheers.

[mschwendt]

That's multiple problems, and if I were you, I would not focus on the slowness first but find out whether you may need to repair your RPM DB further or again. Especially with your ^C activities which are dangerous. You've mentioned "rpm -vv --rebuilddb", does it complain in any way? There are lots of threads to find via Google that discuss RPM DB problems like this.

There's been a thread on fedora users list recently, giving hints on what else you can clean up related to Yum (e.g. yumdb, yum history, yum-plugin-local): http://lists.fedoraproject.org/piper...ad.html#421940

[sevensevenseven]

mschwendt,

Appreciate your reply. I attempted to rebuild the RPM DB again, cleaned yum cache but nothing helped. Figured I'd stop wasting time so I backed up my home directory and reinstalled from scratch... same problem, though!

I realize that stopping yum with ctrl+C isn't a smart thing to do... but i've been extremely "gentle" with yum/rpm after reinstall. Let it take its time no matter how long it took. I just did a yum update and the machine has been stuck at the following for about 30 minutes.

Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction

It paused a few times throughout the update process for 10-30 seconds which is fine.

Whenever yum does this, it's taking up all of the disk IO. Literally close to 100% of the IO for the duration of the "pause". I'm on an SSD.

1903 be/4 root 11.52 K/s 8.64 M/s 0.00 % 80.28 % python /bin/yum update

Thanks for any additional input.

[marko]

Having the same issues even after reinstalling suggests to me your SSD is getting defective, have you ever checked it with smartctl?
(smartctl is in package smartmontools)

smartctl -a /dev/sda

then look if the general health was "PASSED" :

SMART overall-health self-assessment test result: PASSED

and then look at stats related to errors in the disk (this is my report from my spinning disk,
your SSD will look a bit different (SSD's don't have a "spin up time" and so on):

Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 085 085 011 Pre-fail Always - 5370
4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 662
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 0
9 Power_On_Hours 0x0032 095 095 000 Old_age Always - 23339
10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 099 099 000 Old_age Always - 657
13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 099 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 073 071 000 Old_age Always - 27 (Min/Max 16/27)
194 Temperature_Celsius 0x0022 073 070 000 Old_age Always - 27 (Min/Max 16/27)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 6166
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0
201 Soft_Read_Error_Rate 0x000a 253 253 000 Old_age Always - 0

Having a few Hardware_ECC_Recovered is normal but any of the bold ones being non-zero might show a problem

[mschwendt]

Originally Posted by marko

Having the same issues even after reinstalling suggests to me your SSD is getting defective

When it happens so frequently, I second that. Or any other hardware component leading to run-time instabilities.

[sevensevenseven]

Haven't heard of smartctl... Gave it a run and got the following:

sudo smartctl -a /dev/sda
smartctl 5.43 2012-06-30 r3573 [i686-linux-3.4.4-5.fc17.i686.PAE] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Intel 520 Series SSDs
Device Model: INTEL SSDSC2CW240A3
Serial Number: CVCV2024052F240CGN
LU WWN Device Id: 5 001517 bb27f7e41
Firmware Version: 400i
User Capacity: 240,057,409,536 bytes [240 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ACS-2 revision 3
Local Time is: Wed Jul 18 12:47:24 2012 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 2097) seconds.
Offline data collection
capabilities: (0x7f) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Abort Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 48) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x0021) SCT Status supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours_and_Msec 0x0032 000 000 000 Old_age Always - 895699h+08m+19.330s
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 101
170 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 101
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 101
225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 132612
226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 65535
227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 11
228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 65535
232 Available_Reservd_Space 0x0033 100 100 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 100 100 000 Old_age Always - 0
241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 132612
242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 16727
249 NAND_Writes_1GiB 0x0013 100 100 000 Pre-fail Always - 2111

SMART Error Log not supported
SMART Self-test Log not supported
Note: selective self-test log revision number (0) not 1 implies that no selective self-test has ever been run
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I'd be surprised if the disk went bad so quickly. The laptop/SSD is less than 3 months old. Overall health is PASSED but I'm reading docs now to see exatly what the values mean...

[marko]

Originally Posted by sevensevenseven

Haven't heard of smartctl... Gave it a run and got the following:


I'd be surprised if the disk went bad so quickly. The laptop/SSD is less than 3 months old. Overall health is PASSED but I'm reading docs now to see exatly what the values mean...

Those look okay to me, a quick one to check is the Media Wearout Indicator (for SSD) which should be 0 (the digit at the very end after the dash )

Another good stat is that the Reallocated_Sector_Ct is 0

See also:
http://serverfault.com/questions/282...d-drive-health