Bloc d’en Gerard Farràs

See Duncan’s message called “MediaAria2c merged” from zypp-devel mailing list.

Thank you very much Duncan!!

Duncan (my “another mentor” in my Google Summer of Code project), proposed to me to try another approach to solve the failover concept within libzypp library. This time, I will try to create “another” MediaHandler class than will use an external program (aria2c) to download files from download.opensuse.org server. Thus, it will not be required to download metalink file, parse it and download selected package: aria2c will do all of this for us

Extracted from aria2c webpage:

aria2 is a utility for downloading files. The supported protocols are HTTP(S), FTP, BitTorrent (DHT, PEX, MSE/PE), and Metalink. It has a powerful segmented downloading ability, downloading a file from multiple sources and multiple protocols and utilizing your download bandwidth to the maximum. It even supports downloading a file from HTTP(S)/FTP and BitTorrent at the same time, while the data downloaded from HTTP(S)/FTP is uploaded to the BitTorrent swarm.

Using Metalink’s chunk checksums, aria2 automatically validates chunks of data while downloading a file like BitTorrent.

Right now, with libzypp hacked version than I have in my laptop, I can do this:

./zypper install gbrainy
Reading installed packages…

The following NEW package is going to be installed:
gbrainy

Overall download size: 127,0 K. After the operation, additional 472,0 K will be used.
Continue? [YES/no]: YES
Downloading package gbrainy-0.61-31.1.noarch (1/1), 127,0 K (472,0 K unpacked)

——————–
getFile ( /media.1/media )
doGetMetalinkFileCopy –> http://download.opensuse.org/distribution/11.0/repo/oss/media.1/media.metalink
\ Downloading: media.metalink [starting]Metalink file exists – We can parse it
We have 69 mirrors for this file
Downloading from http://mirror.switch.ch/ftp/mirror/opensuse/distribution/11.0/repo/oss/media.1/media
HTTP response: 200
done: /var/adm/mount/AP_0×00000003/media.1/media{- 0644 0/0 size 42}

——————–
getFile ( ./suse/noarch/gbrainy-0.61-31.1.noarch.rpm )
doGetMetalinkFileCopy –> http://download.opensuse.org/distribution/11.0/repo/oss/suse/noarch/gbrainy-0.61-31.1.noarch.rpm.metalink
| Downloading: gbrainy-0.61-31.1.noarch.rpm.metalink [starting]Metalink file exists – We can parse it
We have 69 mirrors for this file
Downloading from http://mirror.switch.ch/ftp/mirror/opensuse/distribution/11.0/repo/oss/suse/noarch/gbrainy-0.61-31.1.noarch.rpm
/ Downloading: gbrainy-0.61-31.1.noarch.rpm [starting]HTTP response: 200
done: /var/adm/mount/AP_0×00000003/suse/noarch/gbrainy-0.61-31.1.noarch.rpm{- 0644 0/0 size 129095}
Downloading:  [done]
Installing: gbrainy-0.61-31.1 [done]

All seems well and correct but… right now begins a new problem If the server responds with HTTP code 304, I get this error message:

Can’t provide ./suse/noarch/gbrainy-0.61-31.1.noarch.rpm : Can’t copy /var/adm/mount/AP_0×00000003/suse/noarch/gbrainy-0.61-31.1.noarch.rpm to /var/cache/zypp/packages/repo-oss
Repository ‘repo-oss’ is out of date. Running ‘zypper refresh’ might help.

Really, this error is caused because this file (/var/adm/mount/AP_0×00000003/suse/noarch/gbrainy-0.61-31.1.noarch.rpm{does not exist}) NOT exists

Post updated 02/07: In effect, we receive 304 response code because we send “If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT” in the header.

GET /distribution/11.0/repo/oss/media.1/media HTTP/1.1
User-Agent: ZYpp 5.0.0 (MediaMetalink – with curl 7.18.1 & libmetalink )
Host: opensuse.mirror.solnet.ch
Accept: */*
If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT

HTTP/1.1 304 Not Modified
Content-Type: text/plain
Accept-Ranges: bytes
ETag: “3592996126″
Last-Modified: Wed, 02 Jul 2008 02:42:38 GMT
Date: Wed, 02 Jul 2008 13:31:06 GMT
Server: httpd/1.4.x

I’m thinking in this actions:

• Disable “If-Modified-Since” in our petition via libcurl, or,
• Try the next mirror when libzypp receives 304 HTTP code response.

Updated: We can disable “If-Modified-Since” header in our petition, simply commenting these lines:

curl_easy_setopt(_curl, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE);
curl_easy_setopt(_curl, CURLOPT_TIMEVALUE, PathInfo(target).mtime());

Good solution ?

Finally, summer is here And, after a lot of rainy days and some exams, my head is thinking in something most appropriated to this time, like:

Go to the beach (21-25 june) , go to Renens (Switzerland) (chez mon frere (28 June – 6 July) ), and, of course, hack libzypp to try to add Metalink support

For these reasons today I have spent some hours in backup some files and remove (forever) Windows partition in my old (and dusty ) laptop, and install OpenSUSE 10.3 in this free place. Also I’ve installed all the needed software (= my new friends ) : gcc compiler, cmake, subversion client, etc..- I must remember, however, upgrade to OpenSUSE_11.0 new week

So, now, I can tell than I’m ready to travel and to enjoy this Google Summer of Code

Renens, this days (picture taken from http://www.renens.ch/):

Finally, seems than my GSoC project takes a more detailed form

After a lot of discussions about “how” to implement the goal of the project (“Libzypp Download Failover”) via zypp-devel mailing list and #zypp channel at irc.freenode.net, we decided than:

* Libzypp will use “only” metalink files (and not Dinamic mirror lists).

* We can use libmetalink (a metalink parser library written in C by Tatsuhiro) to parse metalink files. (We think than this will be better than use external programs like aria2c).

* To download files, we can use, like right now, libcurl.

And… To devel metalink support within libzypp will be my goal in next weeks