Create Local Repo (Script)

[sea]

Heyas

As i just have moved, i currently lack of internet access (mobile phone is limited to 250mb/month), i had a good look on Glennzos Guide about how to set up a local repo.

To just do installations, the script should work, but requires some lovly testers to confirm it.
Currently i'll have a play to retrieve and intigrate the updates.
Sadly, i currently have either access to internet or to my nas (mounted local repo).

So i'd like to share what should work so far, as always, i deny any responsibility, as you use the scripts i wrote with best intention on your own risk!

#!/bin/bash
#
#	This script is written with best intention to help.
#	However, i deny any responsibility that may or may not
#	occour on your system.
#
#	In a first part, it searches $HOME for iso files,
#	lets you select such one. Be sure to edit the script
#	file previous to executing it and verify its release
#	and archbase.
#	Thank you Glennzo for your guide :)
#
#	In the second part, we are aiming to get the updates for
#	our system, and make the system access the local source.
#
#	Lisence:	GPL v3
#	Created:	2012.07.07
#	Changed:	2012.07.11
#	Author: 	Simon A. Erat (sea)
	script_version=0.4
#
# |	-------------------------------------------------------
# |
# |	Title
	clear
	echo
	echo -e "\t\tsea's Create Local Repo and Updates ($script_version)"
	echo
# |
# |	Check $UID
	#if [ ! 0 = "$UID" ]      # Changed to run by users in sudo.
	#then	echo "Must be root to run this script."
	#		return 1
	#fi
# |
# | Variables
	#
	#	Common
	#
	yc=/etc/yum.conf          # Yum Conf
	outputstyle="less"        # Options:	much, less
	where="local"             # Options:	local, nas
	releasever=17             # \_ These two values are also used
	basearch=x86_64           # /  to create the proper sub paths.
	name=$(cat /etc/system-release|awk '{print $1}')
	#name=Fedora              # You may also use hardcoded naming, 
	                          # to do so, just disable above line
	#
	#	Network Attached Storage (NAS)
	#
	nas_pre="http://$nas_ip"
	nas_ip=192.168.10.110     # Enter the fixed IP of your NAS, or
	                          # its name if its in /etc/hosts or
	                          # you use your own DNS server
	#
	#	Paths
	#
	# The repopath is either the structure on the nas or on the partition to be mounted.
	home="/home/$USER"        # This is to avoid /root as $HOME
	pre_local="file:///mnt"
	mountpath=/mnt/$name$releasever
	customrpms="/mnt/Public/Linux/packages"
	repopath="/Public/localrepos/yum/$name/$releasever/$basearch"
	repofile="$name-$releasever-$basearch-local"    # The .repo extension 
	                                                # will be added later by the script
	#
	#	Prepare path structure
	#
	case $where in
	"nas")		repopath="$nas_pre$repopath"	;;
	"local")	repopath="$pre_local$repopath"	;;
	esac
	
	test ! -d $repopath  && sudo mkdir -p $repopath
	test ! -d $mountpath && sudo mkdir -p $mountpath
	
	
# |
# |	Subs
	repo_custom_rpm() { # "/path/of/repo/base"   [ "/PATH/to/your/rpms" ]
	# Checks if first arguments ends with base, if not it adds it.
	# TODO: crosscheck for updates !
		if [ "" = "$1" ]
		then	echo "Usage: repo_custom_rpm \"/path/of/repo/base\" [ \"/PATH/to/your/rpms\" ]"
				return 1
		else	if [ "base" = "$(basename $1)" ]
				then	repobase="$1"
				else	repobase="$1/base"
				fi
		fi
		if [ "" = "$2" ]
		then	echo "Please type its absolute path and hit enter:"
				read myrpmpath
		else	myrpmpath="$2"
		fi
		if [ -d $myrpmpath ]
		then	mylist=$(find $myrpmpath/ -name *.rpm)
				for entry in $mylist
				do	printf "\r    Copy: $(basename $entry)"
					case "$outputstyle" in
					"less")	cp $entry ${repobase:7}
							;;
					"much")	cp $entry ${repobase:7} && \
								printf ", done..\n" || printf ", failed..\n"
							;;	
					esac
				done
				return 0
		else	echo "Sorry path does not exist, exiting...."
				return 1
		fi
	}
	repo_mount_stuff() { #
	# Select the iso file from a list
	# and mounts it to $mountpath
		#
		#	Select iso
		#
		echo
		echo "Please choose an iso, remember: $name-$releasever-$basearch"
		printf "Looking for iso files... (this may take a while)"
		isolist=$(find $home/  -name $name*.iso) ; printf "\r"
		if [ "" = "$isolist" ]
		then	echo "Sorry, didnt find any iso files in $home"
				return 1
		else	select iso in $isolist;do break;done
		fi
		#
		#	Verify DvD Mountpath & mount
		#
		sudo umount $mountpath
		sudo mount $iso $mountpath && retval="$SUCCESS" || retval="$FAILURE"
		
		sE "Mounting $(basename $iso) to $(dirname $mountpath):" "$retval"
		test "$retval" = "$SUCCESS" && return 0 || return 1
	}
	repo_create_it() { # REPOPATH
	# If the creation of the local repo success, it creates the repofile
	#
		test "" = "$1" && echo "Usage: repo_create_it REPOPATH" && return 1
		echo "Creating local repo: $1"
		echo "This will take a while..."
		createrepo "$1" && retval=ok || retval=fail
		test "$retval" = "ok" && \
			echo "Created: $1/repodata." && \
			return 0 || echo "Could not create $1/repodata."
			return 1
	}
	repo_write_repo() { # SOURCEPATH
	# Writes a local repofile to /etc/yum.repos.d. Checks whether 
	# it ends with "base" or "updates", if neither it adds "base"
		tmp=/etc/yum.repos.d
		tmpdd="$tmp/$repofile"
		src="$1"
		case "$(echo $(basename $src))" in
		"base")		type="base"    ;;
		"updates")	type="updates" ;;
		*)			type="base"
					src="$src/base" ;;
		esac
		
		echo "Writing $tmpdd-$type"
		su -c "cat > $tmpdd-$type.repo << EOF
[$repofile-$type]
Name=$name local $type repository - $releasever $basearch 
baseurl=$src
enabled=1
EOF"
	}
	repo_copy_it() { # SOURCEPATH REPOPATH
	# Sourcepath defines the root of the mounted iso file, eg: /mnt/iso-fedora-18
	# Repopath defines the root for the new repo, eg: /mnt/Public/locrep/yum/Fedora/17/x86_64
		test "" = "$2" && echo "Usage: repo_copy_it SOURCEPATH REPOPATH" && return 1
		sourcep="$1"
		repop="$2" ; repop=${repop:7}
		#
		#	List of files
		#
		echo
		echo "Generating file list..."
		rawlist=$(find $mountpath/Packages/ -name *.rpm)
		#
		#	Copy files
		#
		echo "Copying files now:"
			thisline=$(for ((i=0;i<=$(tput cols);i++));do printf " ";done)
		cd $sourcep
		for file in $rawlist
		do	emptyline="${thisline: $[ ${#file} + 15 ] }"
			printf "\r    Copy file: $(basename $file)$emptyline"
			#
			#	Each file on a line, or one line for all?
			#
			case $outputstyle in
			"much") sudo cp $file $repop && \
					printf "Done\n" || \
					printf "Failed\n"	;;
			"less")	sudo cp $file $repop	;;
			"*")	echo "Wrong outputstyle, either 'much' or 'less'" ;;
			esac
		done
		printf "\n"	
	}
	repo_add_updates() { #
	# Trying to get the updates
	# rsync and reposync to check on new system.
		cd $repopath
		reposync --repoid=updates
	}
	repo_toggle_local() { #
	# Simply checks if the word local is found in yum.conf
	# If not, it adds it, if so, it overwrites with 'original' settings.
		current=$(grep "local" $yc)
		if [ "" = "$current" ]
		then	sudo su -c "cat $yc.local >> $yc" && \
					echo "Set to use: local"
		else	sudo su -c "cat $yc.online > $yc" && \
					echo "Set to use: online (orginal)"
		fi
	}
	repo_write_local() { #
	# Writes /etc/yum.conf.local
	# And creates a an yum.conf.online
		#mirrorlist=http://fedora.redhat.com/download/mirrors/updates-released-fc$releasever
		echo "Writing $yc.local"
		su -c "cat > $yc.local << EOF
[base]
name=Fedora $releasever - $basearch - Base
baseurl=$repopath/base
enabled=1

[updates-released]
name=Fedora $releasever - $basearch - Updates
baseurl=$repopath/updates
enabled=1
EOF

cat $yc > $yc.online"
	}
# |
# | Output
	#
	#	Menu Variables
	#
	AllinOne="Full automatic"
	CustomRPM="Add custom RPM"
	MountISO="Mount a downloaded ISO"
	CopyFiles="Copy files from mounted ISO"
	CreateIt="Create Local Repo"
	WriteYC="Prepare yum configuration"
	ToggleYC="Toggle [online|local] yum config"
	AddUpdates="Add local update repo, using rsync"
	WriteRepo="Write a repo file to /etc/yum.repos.d"
	#
	#	Menu Display
	#
	select menu in Back "$AllinOne" "$CustomRPM" "$MountISO" "$CopyFiles" "$CreateIt" "$WriteRepo" "$WriteYC" "$ToggleYC" "$AddUpdates"
	do	case $menu in
		"Back")			break	;;
		"$AllinOne")	if repo_custom_rpm $repopath $customrpms
						then	echo "Added custom repos"
						fi
						if repo_mount_stuff 
						then	echo "mounted to $mountpath"
						else	echo "Maybe file is broken: $iso"
						fi
						if repo_copy_it $mountpath ${repopath:7}/base
						then	echo "Copied files to $repopath"
						else	echo "Maybe $repopath is read-only?"
						fi
						if repo_create_it $repopath
						then	echo "Grats, you got your local repo!"
						else	echo "Sorry, some error occoured :("
						fi
						echo "Gotta TODO the updates part..."
						;;
		"$CustomRPM")	repo_custom_rpm ${repopath:7}/base $customrpms
						;;
		"$MountISO")	repo_mount_stuff 
						;;
		"$CopyFiles")	repo_copy_it $mountpath $repopath
						;;
		"$CreateIt")	case "$(basename $repopath)" in
						"base") 	repo_create_it ${repopath:7} ;;
						"updates")	repo_create_it ${repopath:7} ;;
						*)          repo_create_it ${repopath:7}/base ;;
						esac
						;;
		"$WriteRepo")	repo_write_repo $repopath/base
						;;
		"$WriteYC") 	repo_write_local
						;;
		"$ToggleYC")	repo_toggle_local
						;;
		"$AddUpdates")	echo "NOT WORKING YET!"
						repo_add_updates
						repo_create_it $repopath/updates
						;;
		esac
	done

If you have further suggestions for the updates, please let me know, i'll try to implement it.
Currently looking at rsync (need to know how to get remote path structure), or reposync (wont work at this momen as my original repos are removed).

Thank you and hope it helps you.
Cheerio

[Adunaic]

Currently looking at rsync (need to know how to get remote path structure), or reposync (wont work at this momen as my original repos are removed).

Not sure what you mean here. Do you want to rsync say the fedora-updates repo? That is relativly simple.

[sea]

Originally Posted by Adunaic

Currently looking at rsync (need to know how to get remote path structure), or reposync (wont work at this momen as my original repos are removed).

Not sure what you mean here. Do you want to rsync say the fedora-updates repo? That is relativly simple.

Thats exactly what i'm trying, well, if lucky, once a week for less than 10 minutes.
Could you please give me a hint to the right direction, because with the code using, all i get are the repodata, not no files are downloaded, which makes me think its not working as it should.

Figured there might be a problem, allthough it *should* work as normal user, with the creation of either the repo, or the custom yum.conf.
Please take note, that i do not call it "done" just yet, but need feedback under wich circumstances its working or not.

[Adunaic]

I have not setup a repo sync for F17 yet, so my script is F16.

#rsync.mirrorservice.org/download.fedora.redhat.com/pub/
#rsync://ftp.informatik.uni-frankfurt.de/fedora

echo -e " ############# fedora 16 updates ############# "
rsync --exclude=debug* -av rsync://ftp.informatik.uni-frankfurt.de/fedora/updates/16/i386 /mnt/media/yum/fedora-16/updates/
echo -e " ############# fedora 16 releases ############# "
rsync --exclude=debug* -av rsync://ftp.informatik.uni-frankfurt.de/fedora/releases/16/Everything/i386/os/ /mnt/media/yum/fedora-16/base/

echo -e " ############# fedora 16 - rpm fusion free releases ############# "
rsync --exclude=debug* -av rsync://download1.rpmfusion.org/rpmfusion/free/fedora/releases/16/Everything/i386/os/ /mnt/media/yum/fedora-16/rpmfusion-free
echo -e " ############# fedora 16 - rpm fusion free updates ############# "
rsync --exclude=debug* -av rsync://download1.rpmfusion.org/rpmfusion/free/fedora/updates/16/i386/ /mnt/media/yum/fedora-16/rpmfusion-free-updates

echo -e " ############# fedora 16 - rpm fusion non-free releases ############# "
rsync --exclude=debug* -av rsync://download1.rpmfusion.org/rpmfusion/nonfree/fedora/releases/16/Everything/i386/os/ /mnt/media/yum/fedora-16/rpmfusion-nonfree
echo -e " ############# fedora 16 - rpm fusion non-free updates ############# "
rsync --exclude=debug* -av rsync://download1.rpmfusion.org/rpmfusion/nonfree/fedora/updates/16/i386/ /mnt/media/yum/fedora-16/rpmfusion-nonfree-updates

Then the repo is added as so:

baseurl=file:///mnt/media/yum/fedora-16/base/

[glennzo]

Guys. Have a look at reposync.

[sea]

Updated, add updates using reposync.
Should work, but dont have the bandwith to test.

Note: probably works best if you manualy move /etc/yum.repos.d/fed* to /etc/yum.repos.d/bkup
Have to test this when i'm at home.

---------- Post added at 06:41 PM ---------- Previous post was at 02:27 PM ----------

Just curious, am I right assuming that the "updates" of reposync --repoid=updates, are for both 32 and 64bit?
Found several i386, as well as 86_64, packages inside the created directory among those ~6k files, which lead me to that assumption.

[glennzo]

There are 2 separate paths for i686 and x86_64 updates.

[sea]

Ok thank you Glennzo.

[glennzo]

Sea, just so you're aware, reposync for the updates repos is going to bring in 5958 files. That's a lot of downloading and will take some time.

PHP Code:

reposync --repoid=updates --download_path=/home/glenn/myrepo 


If you don't have reposync, install the yum-utils package.

[glennzo]

Hello again Sea. Looking at what is being downloaded at the moment (reposyncing fedora updates) I'd say you were right. There are i686 and x86_64 packages in the updates repository.

[sea]

Yeah, lucky me i was at a friend with a 20mb flat rate inet so it took less than 1 hour for the ~6k files.
However, as i had used the dvd to create a local repo, i cannot install filezilla, xchat2 and other (small) things.
This brings my thoughts of using reposync as well for the 'base'.

As this will be a 'one time thing', imho it might be worth it to reposync the base (i assume thats 17k+ files) rather then the dvd which are 'only' ~3k files.
But, talking about that many files to be downloaded, reposync is easier (havent looked at manpage yet, but need to figure how to 'set' it to be used for another but the running system) rsync supports to continue where it stop, in case of an internet interruption.

Talking about this, and the experience with the script, i think i'll rewrite it once more. So it can be used with reposync or rsync independantly.
The adds up of the downloads is like 10+ downloads, any feedback ready, so i could improve it while rewriting?

@ Glennzo, this makes it kinda difficult to find a 'proper' place in a directory structure that is thought to be compatible with several versions of fedora/rhel/cent/sl.
Given the (assumtioned) fact that the updates contains both archs, i would place it like: ....../fedora/17/updates.
While the "base" will be inside of: ..../fedora/17/$(arch)/base.
Does this sound like a good idea?

EDIT: reseaching with a 5kbs internet over my mobile phone is pretty annoying.
EDIT2: Oh shoot, overseen Aduniacs post, thanks alot ! That'll do the trick to get started with rsync

---------- Post added at 06:57 PM ---------- Previous post was at 02:36 PM ----------

Rewriting has been done, some part of the configuration must be done inside the script.
Other configuration available from the running script, does only take place if rsync is chosen (reposync is default).

PLEASE let me know if it works, or if not, where it fails.
I only have 90mb left to surf the net over my mobile phone so i cannot test it myself

You can download it at bottom of first post of this thread.

#!/bin/bash
#
#	This script is written with best intention to help.
#	However, i deny any responsibility that may or may not
#	occour on your system.
#
#	In a first step it tries to identify your system
#	and create an according path structure as soon
#	the values are verified.
#	After that, it downloads the repos according to the 
#	hardcoded configuration on top inside the script (you may edit it!)
#
#	After everything, base and updates, are downloaded 
#	it creates the local repos and at the end it writes
#	the local repo file to /etc/yum.repos.d
#
#	Please note: you must be root to run the script!
#	If you cannot execute the script, either
#		*	chmod +x rh-local-repo2.sh
#		*	sh ./rh-local-repo2.sh
#
#	Lisence:	GPL v3
#	Created:	2012.07.07
#	Changed:	2012.07.12
#	Author: 	Simon A. Erat (sea)
	script_version=0.6
#
# |	-------------------------------------------------------
# |
# |	Title
	clear;echo
	echo -e "\t\tsea's Create Local Repo and Updates ($script_version)";echo
# |
# |	Check $UID
	if [ ! 0 = "$UID" ]      # Less issues when running as root
	then	echo "Must be root to run this script."
			return 1
	fi
# |
# |	Configuration
	syncstyle="reposync"      # Options:    reposync, rsync
	where="local"             # Options:	local, nas
	test $where = nas && pre="http://" || pre="file://"
# |
# |	Variables
	# Instead of local path, type NAS' name and path: mynasname/Public/localrepos/yum
	repo_root=/mnt/Public/localrepos/yum
	# Please see::	rsync.mirrorservice.org/download.fedora.redhat.com/pub/
	# 				for a list of available mirrors
	rsync_url=ftp.informatik.uni-frankfurt.de
	lst_archs="i386 x86_64"   # Architectures
	yc=/etc/yum.conf          # Yum Conf
	yr=/etc/yum.repos.d       # Yum Repos dir
# |
# |	Subs
	ask() { # "y/n Question"
	# Ask a question and return 0 if answer is yes, 1 otherwise
	# Sometimes brackets are required, other times, they cause failre
		read -n1 -p " $1 (y/n)" answer 
		echo -e "\r $1 (y/n) : $answer"
		case $answer in
		y|o|j|s)    # First letter of the meaning "yes" in these languages:
		            # English, Français, Deutsch, Italiano
		            return 0    ;;
		*)          return 1    ;;
		esac
	}
# |
# |	Get System Info
	#
	#	Name of Linux
	#
	printf "\r\tRetrieving linux name: \t\t"
	name=$(cat /etc/system-release|awk '{print $1}')	;	printf "$name\n"
	#
	#	Release Version
	#
	printf "\r\tRetrieving release version: \t"
	version=$(cat /etc/system-release|awk '{print $3}')	;	printf "$version\n"
	#
	#	Systems architecture
	#
	printf "\r\tRetrieving system arch: \t"
	arch=$(uname -p)	;	printf "$arch\n"
	#
	#	Change any value?
	#
	if ask "Do you want to change a value of these?"
	then    ask "Is $name correct?" ] || \
		    read -p "Please type the new name: " name
	        ask "Is $version correct?" || \
		    read -p "Please type the version (2 digits): " -n2 version
	        ask "Is $arch correct?" || \
		    select arch in $lst_archs;do break;done
	fi
# |
# |	Prepare environment
	echo 
	echo "Creating tree structure..."
	repo_base="$repo_root/$name/$version/$arch/base"
	repo_upd="$repo_root/$name/$version/updates"
	test ! -d $repo_base && mkdir -p $repo_base || echo "$repo_base allready exists..."
	test ! -d $repo_upd  && mkdir -p $repo_upd  || echo "$repo_upd allready exists..."
# |
# |	Get the synchro done
	echo
	echo "Syncing with upstream..."
	echo
	echo "Get a cup of coffee, a cake or go out for a walk..."
	echo "This WILL take a while, up to several hours depending how fast your internet is..."
	echo
	case $syncstyle in
	"reposync") echo "Working with: $repo_base"
	    cd $repo_base/..
	    reposync --arch=$arch -n --repoid=base .
	    echo "Working with: $repo_upd"
	    cd $repo_upd/..
	    reposync --arch=$arch -n --repoid=updates . 
	    ;;
	"rsync")    echo "Working with: $repo_base"
	    rsync --exclude=debug* -av rsync://$rsync_url/$name/releases/$version/Everything/$arch/os $repo_base
	    echo "Working with: $repo_upd"
	    rsync --exclude=debug* -av rsync://$rsync_url/$name/updates/$version/$arch  $repo_upd
	    ;;
	esac
# |
# |	Create the repos
	echo
	echo "Creating the repos..."
	echo
	echo "Get a cup of coffee, a cake or go out for a walk..."
	echo "This WILL take a while, up to several hours..."
	echo
	createrepo $repo_base
	createrepo $repo_upd
# | 
# |	Create the local repo
	cat > $yr/$name-$version-local.repo << EOF
[base]
name=$name $version - $arch - Base
baseurl=$pre$repo_base
enabled=1

[updates]
name=$name $version - $arch - Updates
baseurl=$pre$repo_upd
enabled=1
EOF
# |
# | Done ?
	echo
	echo "If everything went well, it shoud work now."
	echo "Please let me know at: http://forums.fedoraforum.org/showthread.php?p=1591172"
	echo "Thank you for using a script by sea :)"

[glennzo]

Simon, I'd be happy to test but I'm not so sure I want to get into another multiple hour download session. Are we only working with the updates repo? Do I see a German URL hard coded into the script? I'm in the states.

[sea]

Yeah, its the very same url Adunaic had posted. Since i'm living in a country next to germany, i did use it.
But for the very same reason, just above it there's a comment, asking you to manualy check the mirror list and replace the url
Either way, by default it is using reposync and not rsync, while rsync would use the german url...

No, the new script "2" is again without a menu, as its thought for a one-time-use-set-and-forget.

It'll download the base, of which i assume to be 17k+ files, as well as the updates which are roughly ~6k.
So thats a total of approximatly 23k+ files, after that, i'll create the repodata, and at last the /etc/yum.repos.d/$name-$version-local.repo.

It would be most helpfull if someone who has no local repo yet, but want one, to test the script and repost.
But i guess that wont happen.

---------- Post added at 09:16 PM ---------- Previous post was at 09:16 PM ----------

Yeah, its the very same url Adunaic had posted. Since i'm living in a country next to germany, i did use it.
But for the very same reason, just above it there's a comment, asking you to manualy check the mirror list and replace the url
Either way, by default it is using reposync and not rsync, while rsync would use the german url...

No, the new script "2" is again without a menu, as its thought for a one-time-use-set-and-forget.

It'll download the base, of which i assume to be 17k+ files, as well as the updates which are roughly ~6k.
So thats a total of approximatly 23k+ files, after that, i'll create the repodata, and at last the /etc/yum.repos.d/$name-$version-local.repo.

It would be most helpfull if someone who has no local repo yet, but want one, to test the script and repost.
But i guess that wont happen.