What's the difference between zsync and jigdo? Their core functionality seems the same, so I'm curious on things like performance, file size, and ease-of-use. Would also be interesting to know why one got created when one already existed.
Zsync vs. jigdo
historyperformance
Related Solutions
First, to do general IO testing, I recommend using iozone: http://www.iozone.org/
To properly answer this question, we need more information about the IO subsystems in each server.
However, in general, if you're looking for good IO performance, you need a dedicated hardware RAID card with onboard cache and a battery to back up that cache. This allows the RAID card to perform write-back caching, which can dramatically improve IO performance. The RAID card may also provide better throughput in general compared to onboard controllers.
And finally, an AHCI setting in the BIOS would control an onboard SATA controller. Onboard means it's on the motherboard and is not a server-class standalone hardware RAID card. If IO is not a priority for the workload, a server (whitebox or otherwise) may not have a separate RAID card and may indeed use the onboard controller.
You most assuredly want this BIOS setting set to AHCI, as any other setting will not provide Linux fast, direct access to the drives. If this setting is not making any difference, you might not have any drives connected to the onboard controller, or there may be another misconfiguration which is causing Linux or the BIOS to fall back to IDE-compatibility mode. You can check the kernel boot messages to see what drives the kernel sees and what interface the kernel is using to access those drives.
Now that I look again, I realized you said this was a usb key ( flash drive ) not a hard drive. Flash memory can only be erased in large blocks, and individual sectors can not be written without erasing them ( and the whole block they are in ) first. Since software expects to be able to write wherever it wants on the disk at any time, the disk has translation logic in it to transparently handle the erasing. How this is done has a dramatic affect on write performance. Many devices use an algorithm for most of the disk that handles sequential writes very well, but sucks at random writes. The area near the start of the disk is normally used by the FAT in the FAT filesystem they come preformatted with, and this area is randomly written to frequently, so they use a different algorithm in this area that is slower at sequential writes, but not terrible at random writes.
Thus, I am now pretty sure that my initial guess I added as a comment was right. What you are seeing when you write to the filesystem is the performance of the rest of the disk, and when you dd at offset zero, you are writing to the fat area. If you seek the dd destination a few hundred mb in, it should speed up considerably.
Best Answer
IIRC, Jigdo allows you to break out the files within an ISO and pull the individual files from other servers.
Case in point: Jigdo used to be the preferred way to get Debian ISO files. The Jigdo config file you downloaded would direct the Jigdo client to download all of the files from the regular distribution servers/mirrors, using FTP or HTTP protocols. Then, it would build the ISO client-side. It could start with an existing ISO and "update" it or add stuff which you didn't have time to download and add in, earlier. However, if one of the .deb files (which are compressed) in the ISO changed, you had to download the entire .deb file to rebuild the ISO.
RSync looks at what you have downloaded and what is "current" on the server, downloads the diffs and patches the file(s). Consequently, if you have 650 MB ISO file downloaded, running RSync against it and a server-side version of the ISO file will only download what changed. That allows you to "maintain" a folder or file, relative to another server. That server, however, needs to support the RSync protocol and it will have a pretty high CPU load anytime someone wants to sync with it. Finally, it doesn't work so well for compressed files, because a small change to the uncompressed version can cascade to some VERY significant changes to the compressed version. Consequently, changing a couple files within a .deb file in the ISO will result in downloading the entire new .deb file, not unlike with Jigdo.
ZSync works over HTTP, so no special protocol support is needed. ZSync pushes the CPU load down to the client, instead of the server, so it works better when you have large numbers of users syncing against a centralized server. Finally, small changes to the uncompressed versions of files will result in very small downloads over ZSync, even if the resulting, compressed file has significant changes. If you have an ISO containing thousands of .deb files and you change a few files within one of the .deb files, ZSync would a download just enough information to patch the out-of-date .deb file and re-compress it as needed, then verify the CRC or MD5 signatures, and be done with it. Less bandwidth used, less server-side CPU used, no special protocols needed.
When they can merge all of this with BitTorrent, they've got a winner. When that happens:
That would use even less bandwidth on the server and result in even faster updating. Does anyone know of a protcol/system like that? Last I checked, BitTorrent doesn't allow you to "update" an existing copy of a file.