PDA

View Full Version : I'm writing a script to download third-party stuff...



rkl
19th July 2004, 07:26 AM
One of the main whinges about Fedora Core 2 is that is it's "pure' open source software i.e. no closed source software (or software with non-open licences) is included. Whilst this is fine for the Debian-style purists, it makes life awkward for the rest of us. Time and time again, folks installing Fedora Core have to go through the treadmill of checking for new versions, agreeing to some license or other, downloading it, possibly having to run something to unpack it and even then it's not obvious (e.g. the classic "soft-link in /usr/lib/mozilla/plugins for the Java plug-in or editing /etc/X11/xorg.conf for the Nvidia 3D drivers and so on).

Hence, I've started work on a large shell script that will automate as much as this as possible. BTW, if you just say "set up a yum repository", remember this is for stuff that's mostly closed source and often has some redistribution restrictions on it as well.

Here's the initial output from an early version of the script:



Scanning currently installed packages...
Package Installed Latest Needs Update?
Java SDK 1.4.2_05 1.4.2_05 No
Java JRE 1.4.2_05 1.4.2_05 No
Flash Player 7.0.25 7.0.25 No
Xine Player 040718 040718 No
Xine Codecs 0.52 0.52 No
Xine Library 040718 040718 No
Xine Decryption 1.2.8 1.2.8 No
Opera Browser 7.52 7.52 No
Nvidia 3D Drivers 1.0-6106 1.0-6106 No


It scans the official Web sites for update versions and if there is one, it will pick up the new version number. If it's run as root, it'll also prompt for an update - e.g. here's what happened after I did an "rpm -e opera":



Scanning currently installed packages...
Package Installed Latest Needs Update?
Java SDK 1.4.2_05 1.4.2_05 No
Java JRE 1.4.2_05 1.4.2_05 No
Flash Player 7.0.25 7.0.25 No
Xine Player 040718 040718 No
Xine Codecs 0.52 0.52 No
Xine Library 040718 040718 No
Xine Decryption 1.2.8 1.2.8 No
Opera Browser None 7.52 Yes
Nvidia 3D Drivers 1.0-6106 1.0-6106 No
Update packages (y/n) ? y
Downloading Opera Browser 7.52 RPM (may take a long time)...
Installing Opera Browser 7.52 RPM...
Preparing... ########################################### [100%]
1:opera ########################################### [100%]
Scanning currently installed packages...
Package Installed Latest Needs Update?
Java SDK 1.4.2_05 1.4.2_05 No
Java JRE 1.4.2_05 1.4.2_05 No
Flash Player 7.0.25 7.0.25 No
Xine Player 040718 040718 No
Xine Codecs 0.52 0.52 No
Xine Library 040718 040718 No
Xine Decryption 1.2.8 1.2.8 No
Opera Browser 7.52 7.52 No
Nvidia 3D Drivers 1.0-6106 1.0-6106 No



Obviously, I'll be adding more third-party stuff over time (would like Real Player in there, but not sure if there's any "official" RPMs out there for Real Player 9). And it shouldn't list all the stuff by default (-a option should do that, otherwise there should be some sort of -p option to individually check your favourite packages - you might not want both the Java SDK and JRE installed for instance).

This little script project is really just a way to persuade the FC maintainers to put something like it on their system (only snag is that it relies on the HTML of Web sites not changing their format, so can be fragile if not frequently updated). BTW, did you know that some Opera official mirrors don't have the Linux RPMs on them...tut tut...

Any thoughts on this? Am I wasting my time doing this?

foolish
19th July 2004, 08:52 AM
Looks like you're doing good work here, but maybe it's a bit futile. We already have yum to get packages, and it can easily be configured to get almost anything. The thing yum can't get, it can't get because the owners of the software in question doesn't allow it or because no one has taken the time to create a package for it. I think we need to get working on lobbying for yum repositories instead of just http "one file at a time" downloads from people like Opera, in fact, this thread reminded me and I'm sending them an email now. You should do that to, the more the merrier and so on.

Java is already covered, see www.fedorafaq.org/#java , nvidia is handled by rpm.livna.org most of the time, I guess they're having some trouble now. Xine is covered by yum. See www.fedorafaq.org/#installsoftware on how to install it. Besides, the livna.org or Fedora.us packages are of such high quality, they outshine all semi-official packages from anywhere.

I love the initative and the will to do instead of just complaining, but you're a bit off track here mate. We already have this functionality, if a bit crippled at the moment.

pigpen
19th July 2004, 10:16 AM
I'd appreciate any script that does the dirty work. keep it up!

rkl
19th July 2004, 10:40 AM
We already have yum to get packages, and it can easily be configured to get almost anything.
Not strictly true - there is no graphical interface shipped with yum, nor is there a GUI for modifying yum.conf. Also note that the default /etc/yum.conf does not point to anything other than the standard download.fedora.redhat.com repositories and there are known conflict issues if you try to combine certain yum repositories. It's trivial for those in the know to bring up a terminal and fix these issues, but not for newbies.


The thing yum can't get, it can't get because the owners of the software in question doesn't allow it or because no one has taken the time to create a package for it.
Which is exactly why I wrote the script - yes, use yum repositories where you can, but there's still major exceptions to the rule.


Java is already covered, see www.fedorafaq.org/#java[
Possibly, though I wish Dag's RPM filenames actually matched the Sun Java version (currently 1.4.2_05) ! Also, where's the j2sdk RPM - Dag doesn't have one... :mad:


nvidia is handled by rpm.livna.org most of the time
My script avoids RPMs for NVIDIA drivers because there's way too many combinations if you count kernel and architectures revisions. I just auto-download the installer and run it with the -s flag to automate installation. It also checks for run-level 3 and modifies /etc/X11/xorg.conf or /etc/X11/XF86Config as appropriate to activate the driver.

Xine is covered by yum
Not quite - I get the CVS version that's daily updated, plus the w32codecs RPM, which isn't listed on the usual repositories, so that you can play Windows Media and Quicktime with Xine.


We already have this functionality, if a bit crippled at the moment.
Actually, we don't really have it packaged up for newbies at all really ("severely crippled" would be how I'd put it). If these repositories you've quoted are good enough, why aren't they in the standard yum.conf (even if they're commented out to start with, that's better than nothing)?

I would *love* not to have to write the script I did (only took me a few hours), but current deficiencies in the whole yum/repositories out-of-the-box experience still makes it worthwhile. I just got fed up of doing the same stuff over and over again to get my favourite packages installed, so why not have a single command to get the lot in one go, especially when the versions I want aren't in any of the repositories?

kosmosik
19th July 2004, 10:52 AM
well such script is actually a cool thing. I've got my own ones that do that. my script is rather obscure but works. all it does is:

* remove wicked stuff from the system. by default lots of useless stuff gets installed f.e. in default install you get stuff for operating tape streamers and so on. my script removes that.
* installs and configures apt - since it does it by downloading file from URL then I must go to next step
* update system (including apt)
* reconfigure apt (since updates destroyed previous settings - anybody knows how to go around it? as for now I simply pin apt package and that is it)
* issue few apt commands to install stuff needed
* reconfigure system. f.e. wipe out useless services such as xinetd files (time anyone?), harden security by setting things up and removing few files (f.e. changing su behavior to implict acces only from wheel group, dissallow shutdown etc.)
* set cron jobs and other stuff

generally it is something that helps me with my work. but I think we could set up a project with such things. some of this stuff can be done by anaconda and custom install media but it is too much hassle for me. current thing works well and it is ok :)

kosmosik
19th July 2004, 10:55 AM
uh and I forgot: as for java and nvidia drivers - use packages. people got packages for that do this. Dag's repo has mozilla plugins (think about anything you can get - j2re, acrobat reader etc.), atrpms has nvidia drivers (AFAIR newrpms too), from helix code community you can get plugins for Helix and Real Player. everything as packages - just install it and it will work out-of-the-box. main concern is repos fragmentation and incompatibility :\ without diggin it you'll get f.e. apt reconfiguring itself every update and so on. but it can be done, just requires some work.

rkl
19th July 2004, 11:15 AM
uh and I forgot: as for java and nvidia drivers - use packages.

Yes, where humanly possible, I prefer to install RPMs, because then "rpm -qa" gives you a complete list of what you've got and it's easier to know where the stuff is ("rpm -ql"), what it is ("rpm -qi") and how to upgrade it ("rpm -Uvh").

Having said that, there's always going to be stuff you want that either isn't available as an RPMs or the version out there as an RPM isn't the latest one. Nvidia drivers are indeed awkward because every time there is a driver *or* a kernel update, the RPMs have to be rebuilt - this might explain why Nvidia RPMs seem to be thin on the ground at the moment (I've always used the standalone installer myself).

What I was keen to do with my script was to not have to hack around with yum.conf - just get my favourite packages down (RPMs in all cases except the Nvidia stuff and the Nvidia stuff has to be done separately at run-level 3 anyway) and automate the installation/configuration (yes, sometimes the RPMs don't do all the config'ing they should).

BTW, I am registered on the helixcommunity site - I should go back there and see what's new there to see, for example, if there's a freely downloadable (no reg required) Linux player there that can play Real Audio/Video (not as silly as it sounds - the source-based version doesn't come with Real codecs...).

kosmosik
19th July 2004, 11:48 AM
Having said that, there's always going to be stuff you want that either isn't available as an RPMs or the version out there as an RPM isn't the latest one.
you always can make RPM package yourself :) it is quite easy (issuing one command) with proper tools. also you probably don't need the latest one. all you need it is working secure system. think about it.


Nvidia drivers are indeed awkward because every time there is a driver *or* a kernel update, the RPMs have to be rebuilt - this might explain why Nvidia RPMs seem to be thin on the ground at the moment (I've always used the standalone installer myself).
kernel modules are dependant on kernel version. this sucks - I know. ;\ but in most cases you can grab src RPM and recompile it - it should work in 99% cases (if you always need certain package one hour after new kernel was released), try /sbin/new-kernel-package for automation - this is script launched when new kernel is installed. but you'll need nvirement for compilation on workstations. or just compile it on server and push it to clients...


What I was keen to do with my script was to not have to hack around with yum.conf - just get my favourite packages down (RPMs in all cases except the Nvidia stuff and the Nvidia stuff has to be done separately at run-level 3 anyway) and automate the installation/configuration (yes, sometimes the RPMs don't do all the config'ing they should).
hmm, but when you do all packages by hand you'll have to track every programs change (versioning) and update it manualy. package systems are made just to avoid that.

as for RPMs not configuring things. I don't exactly know what (specific - which package you mean) but most of them works fine. f.e. nvidia drivers (AFAIR, I don't use nvidia nor GLX) in package contain a script that configures X automagicaly - it works like this that you'll never get a system with nonfunctioning X - it configures X on bootup, if driver works with kernel it configures X to work with "nvidia" module, if it is not it makes X use "nv" module...


BTW, I am registered on the helixcommunity site - I should go back there and see what's new there to see, for example, if there's a freely downloadable (no reg required) Linux player there that can play Real Audio/Video (not as silly as it sounds - the source-based version doesn't come with Real codecs...).
hmm I don't know. but if registration is required it can be handled automagicaly by script ;]

rkl
20th July 2004, 02:46 AM
I've added the download of Real Player 10 beta to the script, but the software seems waaay too flaky to me - the standalone player ("realplay") isn't installed into any directory in your PATH which is silly, plus just trying to play the .ram feed of the BBC TV news from the home page of news.bbc.co.uk site caused the player to freeze (which takes down your browser too if you try it via the plug-in).

Quite why they claim it's "beta" when it hangs horribly on a major site's feed like the BBC is anyone's guess - it's barely alpha if you ask me. Looks like Real Player 9 then :(

taylor65
20th July 2004, 03:20 AM
Someone in this thread said there's no GUI for yum, but there is a GUI for apt, which can access the same packages. Check out my how-to on adding packages with a GUI at http://fedoranews.org/contributors/tim_taylor/synaptic/. I think this will do everything you need.

foolish
20th July 2004, 11:53 AM
Introducing a third, or fourth or whatever method of getting files is silly in my oppinion. You are correct, the yum solution isn't perfect. But what's this about graphical interface? does your script have one?

I don't want to discourage good work, because there's not enough of that, and there can never be. But we really don't need more, we need what we already have to work like we want it to. Don't build new stuff, work on existing projects and help them. We don't need more diverson.

But please, keep it up, just don't push this script as any kind of replacement for anything.

rkl
21st July 2004, 09:41 AM
Introducing a third, or fourth or whatever method of getting files is silly in my opinion.
The problem is that yum will *never* be able to be used to get all packages, simply because either they're not available as a repository or there are re-distribution restrictions. Do you have any other solution to getting non-yummable packages other than transferring them from 10 or 20 different sites, one at a time? Nope, you haven't offered one - it's exactly this "apparent niche" that I wanted to fill. But apparently, yum is the only solution you seem to think is available, but I'm pointing out it'll never cover all the situations.

> But what's this about graphical interface? does your script have one?

xterm -e <script> :) Remember, yum doesn't have one either!


But we really don't need more, we need what we already have to work like we want it to.
Quite an idealistic stance - in a perfect world, there'd be no closed source, every package would be a perfect RPM that installed correctly and available for every architecture, plus a yum repository for every known Linux package, a pre-configured yum.conf to include all those repositories and, eventually, a GUI for yum too. The last two are achievable (a GUI for yum in FC3 would be almost reason enough on its own to upgrade to it from FC2 :) ), the rest probably not in our lifetimes.


Don't build new stuff, work on existing projects and help them. We don't need more diversion.
I don't see how I could help the yum software itself do what my script does and I'm quite frankly surprised that you've effectively said "don't bother building a better mousetrap or an alternative mousetrap, just modify an existing mousetrap". Quite against the general Open Source ethos really.


But please, keep it up, just don't push this script as any kind of replacement for anything.
I've never claimed it was - I could say it's "complementary" to yum (it'll get packages that yum can't, which was the raison d'etre to write the script), but I've never claimed it was a replacement for anything (except for the tedious grind of multi-downloads of non-yummable RPMs manually from different sites).

I may not even release the script to the public, BTW, because of its fragile nature (any change in a Web site's HTML and its version/package checking breaks badly - I feel I'd have to version-check the script itself so I could re-distribute new versions of itself when it's run to avoid this breakage issue), so this all might be a moot discussion anyway.

It's a little sad to see a moderator on this FC forum actively discourage one of its members from writing software that doesn't duplicate the functionality of an existing program (it's called "innovation", a term Microsoft bandies around a lot, but never actually does), but that's not stopped me in the past and it won't stop me in the future :)

Update: W00t! The script picked up Opera 7.53 correctly and downloaded/installed it for me :) Strike one for the handy script there...

kosmosik
21st July 2004, 11:41 AM
Introducing a third, or fourth or whatever method of getting files is silly in my oppinion. You are correct, the yum solution isn't perfect. But what's this about graphical interface? does your script have one?

I don't want to discourage good work, because there's not enough of that, and there can never be. But we really don't need more, we need what we already have to work like we want it to. Don't build new stuff, work on existing projects and help them. We don't need more diverson.

But please, keep it up, just don't push this script as any kind of replacement for anything.
well it is not about making next package management system since apt is just fine. it is about to automate stuff that anybody does anyway - installing multimedia and patented stuff... also reconfiguring instalation to need my needs better. just explain me why default desktop instalation includes stuff to operate tape streamers? nobody is using such thigs even on servers... I know it is to hard to change this (installing bloat like this by default) and I have nothing against it. I just make my own scripts and get on with that. some things are not going to be changed (f.e. mp3 support - it is not going into Fedora and that is it - talk with developers) so you have to scratch your own itch...