Bloc d’en Gerard Farràs

See Duncan’s message called “MediaAria2c merged” from zypp-devel mailing list.

Thank you very much Duncan!!

Duncan (my “another mentor” in my Google Summer of Code project), proposed to me to try another approach to solve the failover concept within libzypp library. This time, I will try to create “another” MediaHandler class than will use an external program (aria2c) to download files from download.opensuse.org server. Thus, it will not be required to download metalink file, parse it and download selected package: aria2c will do all of this for us

Extracted from aria2c webpage:

aria2 is a utility for downloading files. The supported protocols are HTTP(S), FTP, BitTorrent (DHT, PEX, MSE/PE), and Metalink. It has a powerful segmented downloading ability, downloading a file from multiple sources and multiple protocols and utilizing your download bandwidth to the maximum. It even supports downloading a file from HTTP(S)/FTP and BitTorrent at the same time, while the data downloaded from HTTP(S)/FTP is uploaded to the BitTorrent swarm.

Using Metalink’s chunk checksums, aria2 automatically validates chunks of data while downloading a file like BitTorrent.

Right now, with libzypp hacked version than I have in my laptop, I can do this:

./zypper install gbrainy
Reading installed packages…

The following NEW package is going to be installed:
gbrainy

Overall download size: 127,0 K. After the operation, additional 472,0 K will be used.
Continue? [YES/no]: YES
Downloading package gbrainy-0.61-31.1.noarch (1/1), 127,0 K (472,0 K unpacked)

——————–
getFile ( /media.1/media )
doGetMetalinkFileCopy –> http://download.opensuse.org/distribution/11.0/repo/oss/media.1/media.metalink
\ Downloading: media.metalink [starting]Metalink file exists – We can parse it
We have 69 mirrors for this file
Downloading from http://mirror.switch.ch/ftp/mirror/opensuse/distribution/11.0/repo/oss/media.1/media
HTTP response: 200
done: /var/adm/mount/AP_0×00000003/media.1/media{- 0644 0/0 size 42}

——————–
getFile ( ./suse/noarch/gbrainy-0.61-31.1.noarch.rpm )
doGetMetalinkFileCopy –> http://download.opensuse.org/distribution/11.0/repo/oss/suse/noarch/gbrainy-0.61-31.1.noarch.rpm.metalink
| Downloading: gbrainy-0.61-31.1.noarch.rpm.metalink [starting]Metalink file exists – We can parse it
We have 69 mirrors for this file
Downloading from http://mirror.switch.ch/ftp/mirror/opensuse/distribution/11.0/repo/oss/suse/noarch/gbrainy-0.61-31.1.noarch.rpm
/ Downloading: gbrainy-0.61-31.1.noarch.rpm [starting]HTTP response: 200
done: /var/adm/mount/AP_0×00000003/suse/noarch/gbrainy-0.61-31.1.noarch.rpm{- 0644 0/0 size 129095}
Downloading:  [done]
Installing: gbrainy-0.61-31.1 [done]

All seems well and correct but… right now begins a new problem If the server responds with HTTP code 304, I get this error message:

Can’t provide ./suse/noarch/gbrainy-0.61-31.1.noarch.rpm : Can’t copy /var/adm/mount/AP_0×00000003/suse/noarch/gbrainy-0.61-31.1.noarch.rpm to /var/cache/zypp/packages/repo-oss
Repository ‘repo-oss’ is out of date. Running ‘zypper refresh’ might help.

Really, this error is caused because this file (/var/adm/mount/AP_0×00000003/suse/noarch/gbrainy-0.61-31.1.noarch.rpm{does not exist}) NOT exists

Post updated 02/07: In effect, we receive 304 response code because we send “If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT” in the header.

GET /distribution/11.0/repo/oss/media.1/media HTTP/1.1
User-Agent: ZYpp 5.0.0 (MediaMetalink – with curl 7.18.1 & libmetalink )
Host: opensuse.mirror.solnet.ch
Accept: */*
If-Modified-Since: Thu, 01 Jan 1970 00:00:00 GMT

HTTP/1.1 304 Not Modified
Content-Type: text/plain
Accept-Ranges: bytes
ETag: “3592996126″
Last-Modified: Wed, 02 Jul 2008 02:42:38 GMT
Date: Wed, 02 Jul 2008 13:31:06 GMT
Server: httpd/1.4.x

I’m thinking in this actions:

• Disable “If-Modified-Since” in our petition via libcurl, or,
• Try the next mirror when libzypp receives 304 HTTP code response.

Updated: We can disable “If-Modified-Since” header in our petition, simply commenting these lines:

curl_easy_setopt(_curl, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE);
curl_easy_setopt(_curl, CURLOPT_TIMEVALUE, PathInfo(target).mtime());

Good solution ?

This week, I’ve been coding “another” MediaHandler class (called MediaMetalink and based from actual MediaCurl), wich first fetchs metalink file (generated via “mod_zrkadlo Download Redirector“), parses it using libmetalink library, and tries to download package using the first mirror: This is only one first version, and there are a lot work to improve this system and make it more robust (ex. if metalink files not exists it would use actual system, if first mirror fails, it would try to download the same file using another mirror, and so on….): But I think it’s an important step

On the other hand, yesterday, I upgraded my system to OpenSUSE 11.0 (thank you Dr. Poeml for your DVD ISO image): There are a lot of new changes in this new version (I think it’s not necessary to comment here these improvements, because I can’t add value to the pile of other sites that explains this better ): However I can see than in this new version, cmake fails when it try to generate Makefile in libzypp branch. This morning I hope I can fix this

This afternoon I will go to Cambrils (Catalan beach), until tuesday, and I will be completly offline: I think it will be fine a few holiday’s loading batteries under the sun, after weeks with a lot of work

Finally, summer is here And, after a lot of rainy days and some exams, my head is thinking in something most appropriated to this time, like:

Go to the beach (21-25 june) , go to Renens (Switzerland) (chez mon frere (28 June – 6 July) ), and, of course, hack libzypp to try to add Metalink support

For these reasons today I have spent some hours in backup some files and remove (forever) Windows partition in my old (and dusty ) laptop, and install OpenSUSE 10.3 in this free place. Also I’ve installed all the needed software (= my new friends ) : gcc compiler, cmake, subversion client, etc..- I must remember, however, upgrade to OpenSUSE_11.0 new week

So, now, I can tell than I’m ready to travel and to enjoy this Google Summer of Code

Renens, this days (picture taken from http://www.renens.ch/):

Finally, seems than my GSoC project takes a more detailed form

After a lot of discussions about “how” to implement the goal of the project (“Libzypp Download Failover”) via zypp-devel mailing list and #zypp channel at irc.freenode.net, we decided than:

* Libzypp will use “only” metalink files (and not Dinamic mirror lists).

* We can use libmetalink (a metalink parser library written in C by Tatsuhiro) to parse metalink files. (We think than this will be better than use external programs like aria2c).

* To download files, we can use, like right now, libcurl.

And… To devel metalink support within libzypp will be my goal in next weeks

Crec que a aquestes alçades la majoria del estudiants del Google Summer of Code ja deuen haver rebut el “misteriós” regal que obsequiava Google, així que ja puc desvelar-ho públicament .

Es tractava del llibre “Beautiful Code” d’O’Reilly.

No coneixia el llibre encara que… sembla força interessant i, de fet, força estudiants ja el coneixien o el tenien. Veig que el llibre disposa també d’una pàgina web i que els drets d’autor es donen a Amnistia Internacional.

Thank you Google!

En Peter, el mentor per al projecte del Google Summer of Code, em va proposar un primer exercici per anar aprenent millor el sistema a desenvolupar.

Es tracta d’un programa en C que faci servir les llibreries CURL per a descarregar un fitxer a través dels repositoris de programari d’openSuse amb la capçalera “Accept-Features: mirrorlist-txt” activada.

Si la resposta del servidor web conté el fitxer en concret, doncs el desem al disc.

Si la resposta del servidor web, en canvi, conté un llistat de mirrors, doncs intentem descarregar el fitxer d’algun d’aquests mirrors.

D’aquesta manera implementem el concepte de failover en la descàrrega de fitxers.

El resultat:

* Codi del programa en C.

* Fitxer Makefile.

Sé que el programa és molt millorable i que conté algun forat en seguretat flagrant, però del que es tractava aquí era solament de fer una prova de concepte i de jugar una mica amb les llibreries CURL.

El següent exercici que em proposa és modificar la llibreria libzypp, afegint en totes les petcions de descàrregues la capçalera: “Accept-Features: mirrorlist-txt”, compilar-la i recompilar altra vegada els clients que la facin servir (en concret “zypper“).

He rebut avui un correu provinent de Google que contenia el text següent:

Congratulations! This email is being sent to inform you that your
application was accepted to take part in the Summer of Code. Please
check your student home page in the SoC web application at
http://code.google.com/soc/student_home.html to determine which of
your applications was accepted.

Així doncs… el projecte ha estat acceptat!

Més informació sobre la tasca a desenvolupar per aquí: Concept for libzypp doing failover when downloading packages from download.opensuse.org