Odd dd behaviour to USB key.

[feddy]

HI, I am trying to prepare a USB to install Fedora to a laptop.

The history of the key is unknown, so it could have a fair bit of use and be showing signs of wear.

bash-4.4#dd if=Fedora-KDE-Live-x86_64-29-1.2.iso  of=/dev/sdi1
^C^Z^X

[1]+  Stopped                 dd if=Fedora-KDE-Live-x86_64-29-1.2.iso of=/dev/sdi1

bash-4.4#dd if=Fedora-KDE-Live-x86_64-29-1.2.iso  of=/dev/sdi1 status=progress
1515807232 bytes (1.5 GB, 1.4 GiB) copied, 438 s, 3.5 MB/s 
dd: writing to '/dev/sdi1': No space left on device
2963457+0 records in
2963456+0 records out
1517289472 bytes (1.5 GB, 1.4 GiB) copied, 529.216 s, 2.9 MB/s

bash-4.4#dd if=/dev/zero  of=/dev/sdi1 status=progress
1882915328 bytes (1.9 GB, 1.8 GiB) copied, 448 s, 4.2 MB/s 
dd: writing to '/dev/sdi1': No space left on device
3680257+0 records in
3680256+0 records out
1884291072 bytes (1.9 GB, 1.8 GiB) copied, 512.604 s, 3.7 MB/s

bash-4.4#dd if=/dev/zero  of=/dev/sdi1 status=progress
1883668992 bytes (1.9 GB, 1.8 GiB) copied, 443 s, 4.3 MB/s 
dd: writing to '/dev/sdi1': No space left on device
3680257+0 records in
3680256+0 records out
1884291072 bytes (1.9 GB, 1.8 GiB) copied, 505.993 s, 3.7 MB/s

So after a manually aborted dd, I restarted and got an incomplete dd write ( 1.5GB out of 1.8 before drop out ).
A subsequent write of zeros got nearer the full size but still dropped out.

The oddity is that the successful writes are exactly the same byte count as the Fedora ISO I originally tried to write.

If I try to write with a more sensible block size it stalls at the same point.

dd if=/dev/zero  of=/dev/sdi1 bs=4k status=progress
1879015424 bytes (1.9 GB, 1.7 GiB) copied, 159 s, 11.8 MB/s
dd: error writing '/dev/sdi1': No space left on device
460033+0 records in
460032+0 records out
1884291072 bytes (1.9 GB, 1.8 GiB) copied, 225.983 s, 8.3 MB/s

How did the failed iso copy manage to reset the "size" of the USB device? This is not first time I've had this kind of oddity. Can anyone explain this behaviour?

Thanks.

[feddy]

The last two plugin events fail to create any devices.

It seems there were 4x 2GB "devices" here , I thought it was a straight 8GB device, so it looks like the out of space was near to but less than capacity of the first "partition".

[164565.169749] scsi 7:0:0:0: Direct-Access     Lexar    USB Flash Drive  1100 PQ: 0 ANSI: 0 CCS
[164565.174958] sd 7:0:0:0: Attached scsi generic sg9 type 0
[164567.464831] sd 7:0:0:0: [sdi] 15663104 512-byte logical blocks: (8.02 GB/7.47 GiB)
[164567.956986] sd 7:0:0:0: [sdi] Write Protect is off
[164567.957004] sd 7:0:0:0: [sdi] Mode Sense: 43 00 00 00
[164568.448980] sd 7:0:0:0: [sdi] No Caching mode page found
[164568.448997] sd 7:0:0:0: [sdi] Assuming drive cache: write through
[164568.458540]  sdi: sdi1 sdi2 sdi3
[164568.463908] sd 7:0:0:0: [sdi] Attached SCSI removable disk
[164569.162786] ISO 9660 Extensions: Microsoft Joliet Level 3
[164569.165275] ISO 9660 Extensions: Microsoft Joliet Level 3
[164569.167178] ISO 9660 Extensions: RRIP_1991A
[174105.721612] usb 1-2: USB disconnect, device number 9
[174113.055166] usb 1-2: new high-speed USB device number 10 using ehci-pci
[174113.187027] usb 1-2: New USB device found, idVendor=05dc, idProduct=a788, bcdDevice=11.00
[174113.187037] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[174113.187044] usb 1-2: Product: USB Flash Drive
[174113.187050] usb 1-2: Manufacturer: Lexar
[174113.187055] usb 1-2: SerialNumber: 75YPHW2IYKG179ZZSD1X
[174113.191277] usb-storage 1-2:1.0: USB Mass Storage device detected
[174113.192269] scsi host7: usb-storage 1-2:1.0
[174114.633973] scsi 7:0:0:0: Direct-Access     Lexar    USB Flash Drive  1100 PQ: 0 ANSI: 0 CCS
[174114.635903] sd 7:0:0:0: Attached scsi generic sg9 type 0
[174116.929349] sd 7:0:0:0: [sdi] 15663104 512-byte logical blocks: (8.02 GB/7.47 GiB)
[174117.421961] sd 7:0:0:0: [sdi] Write Protect is off
[174117.421977] sd 7:0:0:0: [sdi] Mode Sense: 43 00 00 00
[174117.912975] sd 7:0:0:0: [sdi] No Caching mode page found
[174117.912993] sd 7:0:0:0: [sdi] Assuming drive cache: write through
[174117.926902] sd 7:0:0:0: [sdi] Attached SCSI removable disk
[174162.864378] usb 1-2: USB disconnect, device number 10
[174168.235161] usb 1-2: new high-speed USB device number 11 using ehci-pci
[174168.366899] usb 1-2: New USB device found, idVendor=05dc, idProduct=a788, bcdDevice=11.00
[174168.366909] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[174168.366916] usb 1-2: Product: USB Flash Drive
[174168.366922] usb 1-2: Manufacturer: Lexar
[174168.366927] usb 1-2: SerialNumber: 75YPHW2IYKG179ZZSD1X
[174168.370345] usb-storage 1-2:1.0: USB Mass Storage device detected
[174168.370991] scsi host7: usb-storage 1-2:1.0
[174169.801988] scsi 7:0:0:0: Direct-Access     Lexar    USB Flash Drive  1100 PQ: 0 ANSI: 0 CCS
[174169.804040] sd 7:0:0:0: Attached scsi generic sg9 type 0
[174172.097343] sd 7:0:0:0: [sdi] 15663104 512-byte logical blocks: (8.02 GB/7.47 GiB)
[174172.589983] sd 7:0:0:0: [sdi] Write Protect is off
[174172.590000] sd 7:0:0:0: [sdi] Mode Sense: 43 00 00 00
[174173.080974] sd 7:0:0:0: [sdi] No Caching mode page found
[174173.080990] sd 7:0:0:0: [sdi] Assuming drive cache: write through
[174173.092903] sd 7:0:0:0: [sdi] Attached SCSI removable disk

OK, it failed to automount ( no FS ) but there is a valid device.

bash-4.4#time dd if=Fedora-KDE-Live-x86_64-29-1.2.iso  of=/dev/sdi status=progress  bs=4k
1880182784 bytes (1.9 GB, 1.8 GiB) copied, 178 s, 10.6 MB/s
460032+0 records in
460032+0 records out
1884291072 bytes (1.9 GB, 1.8 GiB) copied, 259.5 s, 7.3 MB/s

real	4m19.507s
user	0m0.312s
sys	0m8.689s

OK , look like it completed now I only have one device there. Now to see whether it passes the media check when it boots. ...

[bobx001]

Feddy,

if you are dd'ing an iso onto a stick, you cannot dd it into /dev/sdN1 , you need to dd it into /dev/sdN
cuz the .iso contains the whole partitioning information in itself.

I would though use "mediawriter" (even though they managed to make its GUI almost unusable in some situations, i.e. liveusb-creator was much better).

[feddy]

Thanks Bob, you'll see that's what I did in the last code segement, though that may explain why this initial write to sdi1 set the size of that partition smaller. Not quite sure what would happen writing a partition table to /dev/sdi1 then sending it /dev/zero's . I guess it sets its size to the size of the iso image.

Imagewriter is a mess on LXDE, and I've even seen it hang on windows but the devs seem to think it is someone else's fault.

[marko]

For a really big file like an iso, a 'bs' that's much larger would be better, like bs=1M

What I do right after the dd is done, is remove the key, then reinsert it and immediately do a sha256sum on it:

sha256sum /dev/sdi

Usually that will confirm the same sum as the iso file

[bobx001]

Imagewriter is a mess on LXDE, and I've even seen it hang on windows but the devs seem to think it is someone else's fault.

Yeah, they totally killed it, on Xfce it's also a bloody mess. To get it to work better, what I usually do, is I send the DISPLAY=other_ip to another computer here at home, and X/Windows somehow forces the thing to work properly. IMHO they dereferenced 2 levels of OOP (OLP == Optimistically Lazy Programming), so that now the dialog doesn't even refresh if you move it around or switch desktops lol.

[flyingdutchman]

Do one of these, depending on how much you want to type:
# cat filename.iso > /dev/sdi

or
# dd if=filename.iso of=/dev/sdi bs=1M

It is the same speed, since the underlying I/O system is the same.

[lsatenstein]

Feddy,

if you are dd'ing an iso onto a stick, you cannot dd it into /dev/sdN1 , you need to dd it into /dev/sdN
cuz the .iso contains the whole partitioning information in itself.

I would though use "mediawriter" (even though they managed to make its GUI almost unusable in some situations, i.e. liveusb-creator was much better).

I made a small script that I use with DD.
I am posting it here. It takes a device letter and the ISO. It also asks you to to confirm writing
Without arguments it shows you the full list of devices, to allow you to confirm that your USB is on the appropriate drive letter

#!/bin/bash
# Author Leslie Satenstein 2019-01-12 
# mydd.sh Licensed as gpl3 
# Yours to modify to your hearts content.
# Not responsbile for your goofups
# If necessary, modify ALLOWABLE_Drives to indicate /dev/sd? range. 
ALLOWABLE_Drives="bcdef"  

DRIVELETTER=$1
ISOFILE=$2
rc=0
echo -e "\n$(basename $0 ) version 0.9 12Jan2019"
if [[ $# -ne 2 ]];
then
   echo
   echo  "Use to create Bootable USB. Use as: $(basename $0) singledriveletter  path_to_iso/isofile.iso"
   echo "$(basename $0) First  parameter: drive letter from $ALLOWABLE_Drives"   
   echo "$(basename $0) Second parameter: path_to_iso/isofile.iso"
   echo " You must be certain that the /dev/sd? is for a flashdrive]"
   echo " If in doubt, review below list with sudo lsblk . Reconfirm with lsusb"
   echo
   sleep 3
   echo "List of potential devices to determine drive letter"
   sudo lsblk 
   echo "Repeat: First parameter: drive letter from one of bcdef"
   echo "        Second parameter: path_to_iso/isofile.iso"

   exit 1
fi
SIZE=${#DRIVELETTER}
if [[ 1 != $SIZE ]];
  then
  echo "Only one drive letter please!, Not $DRIVELETTER" >&2
  rc=1
fi

if [[ $ALLOWABLE_Drives != *"$1"* ]]; then
  echo "$1 not one of $ALLOWABLE_Drives "   >&2
  rc=1
fi

# -- iso file should end in dot iso --
if [[ $ISOFILE != *.iso ]];then echo "$2 is not an *.iso file" >&2
   rc=1
fi

if [ ! -e $ISOFILE ];then echo "$ISOFILE does not exist!" >&2 ; rc=1; fi
if [ $rc != 0 ]; then    exit 1 ; fi

DEVICE="/dev/sd$DRIVELETTER"
echo "sudo dd of=$DEVICE if=$2  bs=8M status=progress oflag=direct,sync"
yn="x"
while [[ $yn == "x" ]]
do
   echo -e "continue? y/n "
   read yn
   if [[ "x$yn" != "x" ]];
   then
     if [ "$yn" == "n" ];then exit 0; fi
     if [ "$yn" == "y" ]; then break; fi
   fi
   yn="x"
done

sudo dd if=$2 of=$DEVICE  bs=8M status=progress oflag=direct,sync
echo "eject $DEVICE"

I have 5 Drives on my system. Most of the time the usb is the f drive, but occasionally it is the g drive.

[feddy]

Yeah, they totally killed it, on Xfce it's also a bloody mess. To get it to work better, what I usually do, is I send the DISPLAY=other_ip to another computer here at home, and X/Windows somehow forces the thing to work properly. IMHO they dereferenced 2 levels of OOP (OLP == Optimistically Lazy Programming), so that now the dialog doesn't even refresh if you move it around or switch desktops lol.

Thanks for the info. I thought it was maybe just a corner case on super light LXDE but what you report is exactly the issues I was having.

[alphacrucis]

The fedora documentation recommends using Fedora Media Writer, which is in the fedora repo. The package is called mediawriter. I haven't tried it myself but maybe worth a try.

[bobx001]

The fedora documentation recommends using Fedora Media Writer, which is in the fedora repo. The package is called mediawriter. I haven't tried it myself but maybe worth a try.

This is the utility with the "borked" interface we are talking about with feddy.