Reply
 
Thread Tools Display Modes
  #1  
Old 09-20-2006, 07:10 PM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default Easier downloads / improved download pages / metalink
Now that the aria2 PBI is available, I hope people can try out metalinks for the PC-BSD ISO large downloads. The metalinks list all the mirrors and checksums for a download, so the download can be much faster and more reliable.

From the command line, you would use it like:
Code:
aria2c http://www.metalinker.org/samples/PC...1.iso.metalink
Amigamerlin might work on a GUI for it, so it will be even easier for people to use.

(I also want to do a PBI for another GUI metalink client, but it's based on wxWidgets/wxWindows and GTK so that might take some time).

The three metalinks are at

http://www.metalinker.org/samples/PCBSD ... o.metalink
http://www.metalinker.org/samples/PCBSD ... o.metalink
http://www.metalinker.org/samples/PC-BS ... p.metalink

Here's what one looks like, XML text, but the user never sees this. They just use this link. The info can be generated automatically, which makes things easier.
Code:
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" generator="Metalink Generator v1.00.0034" xmlns="http://www.metalinker.org/">
  <publisher>
    <name>PC-BSD</name>
    <url>http://www.pcbsd.org/</url>
  </publisher>
  <description>PC-BSD has been designed with the "casual" computer user in mind. Installing the system is simply a matter of a few clicks and a few minutes for the installation process to finish. Hardware such as video, sound, network and other devices will be auto-detected and available at the first system startup. Home users will immediately feel comfortable with PC-BSD's desktop interface, with KDE 3.5 running under the hood. Software installation has also been designed to be as painless as possible, simply double-click and software will be installed.</description>
  <files>
    <file name="PCBSD-1.2-x86-CD1.iso">
      <version>1.2</version>
      <os>BSD-x86</os>
      <verification>
        <hash type="md5">7db7864465651407e4e10bcea47542ad</hash>
      </verification>
      <resources>
        <url type="http" location="us" preference="100">http://mirror2.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="gr" preference="100">http://mirror3.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="no" preference="100">ftp://mirror4.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="tw" preference="100">ftp://mirror5.pcbsd.org/PCBSD/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="it" preference="100">http://mirror6.pcbsd.org/1.2/PCBSD-1.2-x86-CD1.iso</url>    
        <url type="ftp" location="jp" preference="100">ftp://mirror7.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="de" preference="100">http://mirror8.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="tw" preference="100">ftp://mirror9.pcbsd.org/pub3/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="lv" preference="100">ftp://mirror10.pcbsd.org/pub/PCBSD/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="uk" preference="100">http://mirror11.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="us" preference="100">http://mirror12.pcbsd.org/distros/pc-bsd/i386/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="uk" preference="100">ftp://mirror13.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="it" preference="100">ftp://mirror14.pcbsd.org/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="br" preference="100">ftp://mirror15.pcbsd.org/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="ro" preference="100">ftp://mirror16.pcbsd.org/pub/PCBSD/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="de" preference="100">ftp://mirror17.pcbsd.org/pub/Mirrors/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="se" preference="100">ftp://mirror18.pcbsd.org/pub/PC-BSD/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="sk" preference="100">http://mirror19.pcbsd.org/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="it" preference="100">ftp://mirror20.pcbsd.org/mirrors/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="us" preference="100">ftp://mirror21.pcbsd.org/1.2/PCBSD-1.2-x86-CD1.iso</url>
  		<url type="http" location="uk" preference="100">http://mirror22.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="id" preference="100">ftp://mirror23.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="http" location="us" preference="100">http://mirror24.pcbsd.org/mirror/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="ftp" location="bg" preference="100">ftp://mirror25.pcbsd.org/pub/PCBSD/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="ftp" location="ua" preference="100">ftp://mirror26.pcbsd.org/pub/mirrors/www.pcbsd.com/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="ftp" location="ch" preference="100">ftp://mirror27.pcbsd.org/pub/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="http" location="hu" preference="100">http://mirror28.pcbsd.org/1.2/PCBSD-1.2-x86-CD1.iso</url>
		<url type="ftp" location="us" preference="100">ftp://mirror29.pcbsd.org/downloads/mirrors/pcbsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
        <url type="ftp" location="jp" preference="100">ftp://mirror30.pcbsd.org/pub/pc-bsd/1.2/PCBSD-1.2-x86-CD1.iso</url>
          </resources>
    </file>
  </files>
</metalink>
These could also be used on pbidir.com, one link instead of:
Code:
Please select your download location:

    * Mirror 22 (Estonia - eenet.ee)
    * Mirror 21 (USA/Texas - prismnet.com)
    * Mirror 20 (Slovakia - backbone.sk)
    * Mirror 19 (Italy - Garr.it)
    * Mirror 16 (Romania - kappa.ro)
    * Mirror 17 (Germany - esslingen.de)
    * Mirror 18 (Sweden - Dataphone.se)
    * Mirror 2 (USA - isc.org)
    * Mirror 3 (Greece - ntua.gr)
    * Mirror 5 (Taiwan - cs.pu.edu.tw)
    * Mirror 4 (Norway - netrunner.nu)
    * Mirror 6 (Italy - fastbull.org)
    * Mirror 7 (Japan - allbsd.org)
    * Mirror 8 (Germany - plusline.de)
    * Mirror 9 (Taiwan - tnc.edu.tw)
    * Mirror 10 (Latvia - bsd.lv)
    * Mirror 11 (UK - mirrorservice.org)
    * Mirror 19 (Slovakia - Backbone.sk)
    * Mirror 13 (UK - Plig.net)
    * Mirror 14 (Italy - MuEurope.org)
    * Mirror 15 (Brazil - c3sl.ufpr.br)
You can also use backwards compatible links like this:
http://mirror2.pcbsd.org/pub/pcbsd/1.2/ ... o.metalink

Clients that are not metalink aware will use the first URL; Metalink clients will use the second URL.
Reply With Quote
  #2  
Old 09-20-2006, 07:35 PM
TerryP TerryP is offline
Senior Member
 
Join Date: Nov 2005
Location: Ga. USofA
Posts: 7,906
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to TerryP Send a message via AIM to TerryP Send a message via Yahoo to TerryP
Default
All the best tools are born in console me thinks.

File managment/managers
Text editors
File Viewers
Compilers
Debuggers
Help programs
FTP Clients
Download Managers
Web Browsing
E-Mail clients
Word Processing and Type Setting
Classic Games
e.t.c.

All of them have GUI's now in one form or another so I'm sure Aria2 will too.
Reply With Quote
  #3  
Old 09-23-2006, 04:55 AM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default
TerryP, I know what you mean. I love command line tools. For metalink tho, GUI programs came first. aria2 is the only CLI metalink client.

Here's another cool thing about metalink. Besides mirrors & checksums, it stores all sorts of other metadata like OS & language...

Language:
http://www.metalinker.org/samples/Nero- ... l.metalink

This file lists 3 languages (all Windows binaries): en-US, de, it. You would use it like

aria2c http://www.metalinker.org/samples/Nero- ... l.metalink --metalink-language=LANGUAGE

Select the language you want (For example: "--metalink-language=de" for German), and it only downloads that file.

OS:
http://www.metalinker.org/samples/OOo/O ... l.metalink

This file lists binaries for 4 OSes: Windows-x86, Linux-x86, Solaris-x86, MacOSX-PPC.

Use it with:

aria2c http://www.metalinker.org/samples/OOo/O ... l.metalink --metalink-os=OS

Select the OS you want (For example: "--metalink-os=Linux-x86" for that arch), & it only downloads that specific one.

When automatically generated, its easy to create metalinks with every OS/language combo, for say OOo (or anything else). Tie that in with location, & you have one download link for everyone. Someone in Brazil that speaks French with BSD, will click on the link & get the French BSD version from the local Brazilian mirrors. Someone in Japan that speaks English on OS X can click that same link & get the OS X/English version from the local Japanese mirrors...etc

No more mirror lists on web pages, or language, OS selection, or manually verifying checksums.
Reply With Quote
  #4  
Old 09-23-2006, 05:18 AM
TerryP TerryP is offline
Senior Member
 
Join Date: Nov 2005
Location: Ga. USofA
Posts: 7,906
Thanks: 0
Thanked 0 Times in 0 Posts
Send a message via ICQ to TerryP Send a message via AIM to TerryP Send a message via Yahoo to TerryP
Default
Tbh,, it sounds like it should be a standard practice, like using a web browser and http together :-)
Reply With Quote
  #5  
Old 09-24-2006, 05:39 PM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default
Originally Posted by TerryP
Tbh,, it sounds like it should be a standard practice, like using a web browser and http together :-)
Thanks, I agree

It's getting there. OpenOffice.org has been using it for months, & a bunch of Linux & BSD distributions are now too as well. Hopefully PC-BSD will be one of them some day.

A new version of aria2 came out, I updated the PBI so that should be available some time soon.
Reply With Quote
  #6  
Old 09-26-2006, 04:12 PM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default
The new version of Simba, a mirroring tool, is out. It generates metalinks automatically when new files are synced. Check it out in action. It creates great reports & displays em in a nice way. Maybe it could be used on pbidir.com?
Reply With Quote
  #7  
Old 09-30-2006, 05:28 PM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default
Here's how to use aria2 with Firefox:
"Add another download manager in FlashGot, choose '/usr/kde/3.5/bin/konsole' as an executable and put '-e aria2c [URL]' in the arguments text box. This works with KDE, if you'd like to use Gnome terminal search what arguments it needs."

Here's a comparison between a regular download from one server, and getting segments from multiple servers with metalink.

Multithreaded downloads can increase speed quite a bit.

This is from home (6 megabit cable).

The first example downloads the file in one segment. The second example downloads the metalink which lists the mirror locations & checksum of the file, downloads the file in 5 segments from 5 different mirrors, then verifies the checksum - so the actual download is faster than the time listed.

$ time aria2c http://www.kernel.org/pub/linux/kernel/ ... .10.tar.gz
51,552,272/51,709,946 Bytes 99% 00s 152.41 KB/s 1 connections
The download was complete. <./linux-2.6.17.10.tar.gz>

real 5m26.995s
user 0m2.040s
sys 0m13.529s

$ time aria2c http://download.packages.ro/metalink/ke ... z.metalink
0/21,268 Bytes 0% - 0.00 KB/s 1 connections
The download was complete. <./linux-2_6_17_10_tar_gz.metalink>
51,709,946/51,709,946 Bytes 100% 00s 36.77 KB/s 5 connections
The download was complete. <./linux-2.6.17.10.tar.gz>
Now verifying checksum.
This may take some time depending on your PC environment and the size of file.
checksum OK.

real 2m2.496s
user 0m1.288s
sys 0m9.145s
Reply With Quote
  #8  
Old 10-07-2006, 06:58 PM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default
Looks like metalink support is making its way into KGet, so hopefully it will be in KDE4 TP1.
Reply With Quote
  #9  
Old 10-18-2006, 06:45 PM
ant ant is offline
Member
 
Join Date: Sep 2006
Location: Pompano Beach, Florida, USA
Posts: 14
Thanks: 0
Thanked 0 Times in 0 Posts
Default
Metalink tools are now available for easily making metalinks. They generate every conceivable checksum & import mirrors/p2p.
Reply With Quote
  #10  
Old 10-21-2006, 07:11 PM
asarch asarch is offline
Junior Member
 
Join Date: Oct 2006
Posts: 1
Thanks: 0
Thanked 0 Times in 0 Posts
Default
Hi mr. ant:

I've try both samples you post, however, the second one it only downloads me the metalink file itself and not the content of the file.
Code:
$ time aria2c http://download.packages.ro/metalink...ar_gz.metalink
Why? :twisted:
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
man pages jdarnold Creating PC-BSD Packages (.pbi files) 18 08-31-2006 08:26 PM
Surf man pages online TerryP Tips and Tricks 0 07-19-2006 09:42 PM
Easier Disk Formatting Watts Feature Requests 0 02-21-2006 04:38 AM
Getting Ports working- makign it easier to deal with? modmans2ndcoming General Questions 4 02-18-2006 03:46 AM
Konqueror opens pages in Kate Charles General Bug Reports 1 01-20-2006 02:09 AM


All times are GMT. The time now is 08:43 PM.


Powered by vBulletin® Version 3.8.5
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.

Copyright 2005-2010, The PC-BSD Project. PC-BSD and the PC-BSD logo are registered trademarks of iXsystems.
All other content is freely available for sharing under the terms of the Creative Commons Attribution License.