mars 25, 2008

Bernstein's comment about third-party DNS and the reality with qmail

In a document written a couple years ago, Dan Bernstein describes the costs and benefits of third-party DNS services.

In his document, he states that:

``Third-party DNS means the difference between email being requeued and email bouncing.'' No, it doesn't. Mail transfer agents defer delivery attempts when DNS servers are unreachable, just as they defer delivery attempts when SMTP servers are unreachable.

This is simply not true.

If no DNS servers are available, then qmail (written by Bernstein) will bounce the message immediately with a permanent error like this:

Hi. This is the qmail-send program at some.server.
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

<me@foo.bar>:
Sorry, I couldn't find any host named foo.bar. (#5.1.2)

If the domain owners were using third party DNS servers, qmail (or any other SMTP agent) would see that the MX servers are unavailable, and generate a temporary error. If the MX servers are unavailable for several hours, the sender will receive a temporary error message, but the message will not bounce unless the MX servers are unavailable for several days.

Posted by gfk at 11:21 AM | Comments (0) | TrackBack

juin 19, 2007

Hard drive woes

Yesterday at around 16:30 the secondary hard drive on my decade old linux router passed away. Most of the software continued working, only the DNS server stopped working. Not knowing what was going on, I restarted it, and it's not starting anymore... :-( I don't believe in esotericism, but I guess I had a premonition last week-end when I bought a Soekris net4501 that I should be receiving in a day or two. So for those who are emailing me, it might take me a bit more time to answer, since my mail server was behind that router. You can reach me at gfilion@gmail.com in the meantime. Update 20070619 19:57: I managed to use a Linksys wireless router running DD-WRT as the new router. It's not as good as my old one, but it will do until I can set up the Soekris board. Update 20070622: The Soekris board is now set up with m0n0wall with openvpn support. It's a bit more rigid than I had hoped for, but at least it's really easy to set up.

Posted by gfk at 10:25 AM | Comments (0)

février 16, 2007

The rsync algorithm applied to S3

Many people would like to see rsync work with S3, unfortunately S3 is a "dumb" file storage that can't run rsync. It's possible to run rsync on a EC2 that accesses your files on S3, but it's complex, (more) expensive and not elegant.

I spent some time reading the rsync algorithm and seeing how it could be adapted to S3 (or any other dumb file storage like a FTP server).

Suppose we have one general purpose computer α and a dumb file storage service S3. We want to minimize the amount of data transfered between α and S3.

Upload

Computer α wants to place file A on S3.

The modified rsync algorithm consists of the following steps:


  1. α splits the file A into a series of non-overlapping fixed-sized blocks of size S bytes. The last block may be shorter than S bytes.

  2. For each of these blocks α calculates two checksums: a weak "rolling" 32-bit checksum (described in the rsync algorithm) and a strong 128-bit MD4 checksum.

  3. α creates a XML file A' on S3, that contains the checksums calculated in step 2, a whole-file MD5 checksum and the metadata for file A

  4. α compresses and encrypt each block of file A, then stores each of them in a different object on S3. Each of these block-object has its MD4 checksum as its name and is stored in a bucket P. If the block-object already exists in P, there's no need to resend the file.

Download

Computer α wants to get file B from S3 and already has file A where file A and B are "similar."

The modified rsync algorithm consists of the following steps:


  1. α splits the file A into a series of non-overlapping fixed-sized blocks of size S bytes. The last block may be shorter than S bytes.

  2. For each of these blocks α calculates two checksums: a weak "rolling" 32-bit checksum (described in the rsync algorithm) and a strong 128-bit MD4 checksum.

  3. α gets XML file B' from S3.

  4. α searches through A to find all blocks of length S bytes (at any offset, not just multiples of S) that have the same weak and strong checksum as one of the blocks of A. This can be done in a single pass very quickly using a special property of the rolling checksum described in the rsync algorithm.
  5. α gets each block-object listed in B' from P that is does not already has. It then decrypts and decompresses them.
  6. α reconstitutes file B using blocks from file A and blocks downloaded from P.

  7. α validates the integrity of the file by verifying it's checksum listed in B'.

  8. Α restores the file metadata as listed in B'.

Similarity with existing products

With hindsight, this looks a lot like what S3InfiDisk is doing, with the exception that S3InfiDisk uses 1 MB blocks while rsync uses 1 KB blocks.

Posted by gfk at 8:31 AM | Comments (1)

septembre 20, 2006

Bad file descriptor on perl and IIS 6 (Win2003)

Here's a sticky problem that took me quite a while to find out.

I had a perl script that did a qx(something.bat $file); it worked fine on Win2000/IIS5, it worked fine on the command line on Win2003, but on the web site, I would always get a Bad file descriptor error.

So I replaced qx by system, and then I got the error: Can't spawn cmd.exe Then it was easy to figure out that the problem was related to cmd.exe.

The problem was that the IUSR_WEB user did not had the Read/Execute permissions on C:\windows\system32\cmd.exe. Also be sure to put $ENV{'PATH'} = 'C:/windows/system32/;'; on top of your script.

So in case anyone has this problem in the future, that's the way to solve it.

Posted by gfk at 10:55 AM | Comments (1)

mai 18, 2005

Is this the way to Armadillo?

Des petites histoires de l'autre coté de la grande flaque:

En 1971, un crooner du nom de Tony Christie a sortie une toune s'intitulant "Is This The Way To Amarillo?" Amarillo est une petit ville du Texas. C'était une parodie qui faisait semblant de faire la promotion touristique de ce petit bled texan.

Au début 2005, Peter Kay, un humoriste britanique, refait le vidéo de la toune en "lip-sync" avec la collaboration de l'auteur. Dans le vidéo on y voit plein de stars britaniques y faire des apparitions caméo. Cette toune est monté au 1er rang des "top charts" britaniques durant les mois de mars et avril 2005.

Voir le vidéo clip de Peter Kay.

Une parodie de cette repise a été faite par des soldats britaniques stationnés en Iraq, elle s'intitule "Is this the way to Armadillo" (Armadillo est un tatou, la petite bébite). Elle a été tellement téléchargée par les employés du Ministère de la Défense que ça a fait planté leur réseau.

Lire un article de la BBC qui parle du "plantage".

Voir la parodie par les soldats.

Ha, ces chers british, que ferait-on sans eux?

Posted by gfk at 5:48 PM | Comments (0)

mars 28, 2005

My servers

For your inner nerd who wants to know what is under the hood of my servers, here's the place to look.

baba

The site that you're reading this on is hosted on a small virtual private server called baba that is rented from JVDS. It's located somewhere in California. I'm not sure about the hardware it is running on, but from JVDS's description it's a dual Xeon, 2 GB of RAM and hardware RAID1. It's running FreeBSD with these public services:

ali

My main server is called ali, it's located in my basement, and is not directly accessible from the Internet. The hardware is upgraded quite often, but at the last update, it had an AMD Athlon XP2000+ CPU running at 1.3GHz on an ASUS A7V266-C motherboard. It has 512MB of RAM and 80GB of hard disk in a software RAID 1 configuration. The two hard drives are on two separate IDE controllers, one on the motherboard (vt8233a driver) and the other on a PCI controller (Sil680 driver). The boot partition is located on two 256MB compact flashs mounted as hardware RAID 1 using a DupliDisk device. In addition to the CPU and power supply fans, it has two additional fans, a standard 80 mm fan in the front bottom of the case that sucks cold air in -- yes, they only had fans with blue leds in them -- and a Twin Turbo fan that blows the hot air out on front top of the case. Being in the basement, the room temperature is always between 15°C and 20°C. It's running Debian with these private services: baba acts as a private relay to ali through a VPN. If my home connection or ali goes down, baba will queue the messages until ali comes back up. I love and would recommend most of these programs. qmail, tinydns and dnscache are made by prof. Dan Bernstein and are fanstastic, truly the best for the job, despite some "political" issues associated with them and Bernstein. dovecut and vsftpd are pretty awesome too and I beleive that they have been inspired by Bernstein's work. qpsmtpd is a mix qmail and perl, which are pretty much at both ends of the software design spectrum, but despite this -- I would even say because of this -- the results are very good. Nagios is not a very elegant program, and is confusing to use, but it does the job well. The only one that I would not recommend is ISC DHCPD, while it has never failed on me, I have a pretty bad prejudice against anything made by ISC. I looked for an alternative but couldn't find a satisfying one, and since I don't have any evidence that DHCPD is bad, I'm sticking with it for the moment. If you know a good replacement for it, don't hesitate to leave a comment.

Posted by gfk at 4:08 PM | Comments (0)

décembre 28, 2004

Software defined radio

I just read that a Universal Software Radio Peripheral (USRP) has now gone into production. The unit is priced at $450 for the motherboard and about $50 for daughterboards (see the links below for more infos about mother/daughter boards). It's still quite expensive, but considering that the next cheapest one costs tens of thousands of dollars, this one is pretty attractive. Maybe the next version will cost around 100$ and everyone will want one (or two).

Here are places to get more infos about the USRP:
http://comsec.com/wiki?UniversalSoftwareRadioPeripheral
http://www.gnu.org/software/gnuradio/doc/exploring-gnuradio.html#usrp

Another nice thing with this new SDR is that it can receive and decode over-the-air HDTV signals. Even before the broadcast flag becomes mandatory for HDTV receivers, it can be bypassed with the software decoding of the HDTV signal. The broadcast flag is a code sent with the HDTV signal that is supposed to disable the recording of copyrighted works.
See http://www.gnu.org/software/gnuradio/hdtv-samples.html

For people wondering about this SDR and why it's so great, I put some quotes below from a slashdot.org discussion. The entire discussion, including all the usual conspiracy theories that "the government won't allow this" and "the Hollywood industry is controlling Washington" can be read at: http://it.slashdot.org/article.pl?sid=04/12/27/1341221

-----
by stienman (51024) on 27.12.2004 15:30 (#11193892)
-
Of the set of problems in the world that can be solved using software only, hardware only, or a mix of the two, problems are generally moving toward the software solving side.

In other words, radio can be completely received, down converted, and demodulated in hardware and by and large this is how it is done.

However, if you instead receive and downconvert the radio signal, then you can let software take over for the demodulation, and in the case of HDTV further digital decoding.

Further, this device can work on about 32MHz of the signal spectrum at a time. This doesn't mean much until you realize that the entire FM radio band (88.1MHz - 107.9MHz) fits within that slice of bandwidth. You can use this radio to decode the entire audio of all the radio stations in the area simultaneously. Live in detroit? Listen to and record every single radio station with one device. Not so terribly useful for the consumer, but nice for the re-streamer, radio fanatic, FCC, NSA, etc.

Bandwidth of an NTSC TV signal is about 6MHz. Watch/record 4-5 consecutive channels simultaneously.

HDTV is about 8MHz. Watch and record 3-4 consecutive channels simultaneously.

In short, it's a move from less hardware to more softare. The biggest advantage is not less hardware, but more flexibility. This one tuner can be used to tune your HDTV, TV, radio, 802.11, bluetooth, etc.
-----
by pe1chl (90186) on 27.12.2004 15:27 (#11193865)
-
The device is mainly a fast analog-to-digtal and digital-to-analog converter, with USB interface.
It allows you to quickly readout a couple of analog signals using a PC, and to generate some analog signals under program control.

With some additional radio hardware (supplied on daughterboards) you can convert a certain frequency band into analog signals that are then fed to the converters. With proper software you can use this as a radio that does not have a tuning knob but can be tuned in software and/or to receive an entire radio frequency band and process all the signals in parallel.

For example, you can draw a picture of the signal strength for each frequency, plot this versus time. Or you could write software that receives all FM stations in your area simultaneously and converts their broadcasts to streaming audio channels.
-----
by Animats (122034) on 27.12.2004 13:36 (#11192912)
-
This could be valuable for high-bandwidth instrumentation applications. Wideband data-acquisition cards tend to be both overpriced and out of date, because the product volumes are small.

Some years ago, I was doing some work on a laser rangefinder, and got to the point where I needed about $20K in test gear to find out why it wasn't working right. Something like this would have been a big help.

Radio hams will find uses for this. It should be great for working on new data transmission schemes for high-noise links, like HF.
-----

Posted by gfk at 2:33 PM | Comments (0)

décembre 25, 2004

Caméras du MTQ

J'habite à Beaumont et je travaille à Sainte-Foy, je dois donc faire le trajet sur l'autoroute 20 de Beaumont aux ponts deux fois par jour. J'ai souvent recourt aux caméras de circulation du Ministère des transports du Québec (MTQ) pour connaitre les conditions routières sur ce tronçon. J'ai créé une interface plus simple et plus rapide pour voir les images de ces caméras.

Posted by gfk at 3:14 PM | Comments (0)

novembre 12, 2004

Généalogie

J'ai trouvé cette feuille [168 Ko] dans un tiroir, j'ai recopié les données si ça peut intéresser quelqu'un.

Généalogie des Filion au Québec

André Filion
Gabrielle Senler
Saint-Germain-de-l'Auxerrois, Paris, Île-de-France

1ère génération

Antoine Filion, 1666
Anne Anneville
Paris

2e génération

Jean Filion, 6 juin 1695
Françoise Simard-Senat
Québec

3e génération

Paul Filion, 12 novembre 1731
Marie-Josette Tremblay
Baie-Saint-Paul

4e génération

Antoine Filion, 29 octobre 1770
Victoire Girard
Baie-Saint-Paul

5e génération

Pierre Filion, 25 septembre 1798
Marie-Geneviève Lessard
Saint-Joachim

6e génération

Pierre Filion, 13 février 1827
Théetiste Elie Breton
Saint-Joachim

7e génération

Pierre Filion, 21 juillet 1868
Virginie Boucher
Saint-Joachim

8e génération

Sylvio Filion, 19 avril 1909
Blanche Dussault
Victoriaville

9e génération

Émile Filion (dit Jack), 24 décembre 1938
Cécile Gagné
Victoriaville

10e génération

Jacky Filion, 6 août 1946
Nicole Laliberté
Québec

11e génération

Guillaume Filion, 9 août 1980
Beaumont

Posted by gfk at 7:07 PM | Comments (0)