Kodewerx

Our culture has advanced beyond all that you could possibly comprehend with one hundred percent of your brain.
It is currently Thu Dec 05, 2024 10:02 pm

All times are UTC - 8 hours [ DST ]




Post new topic Reply to topic  [ 7 posts ] 
Author Message
 Post subject: Patching Format Idea
PostPosted: Wed Apr 28, 2010 3:43 am 
Offline
Komrade
Komrade

Joined: Tue Mar 27, 2007 10:18 am
Posts: 1328
So I was digging through my doc, and I ran into a text file I had tossed in there a couple of months ago.

The contents of it:

http://pastebin.com/nB5vck7Z

I personally think it's simple and genius.

But at the moment I just don't have the time to mess with the idea.

C&C yo.

_________________
Image


Top
 Profile  
Reply with quote  
 Post subject: Re: Patching Format Idea
PostPosted: Wed Apr 28, 2010 4:18 pm 
Offline
Krew (Admin)
Krew (Admin)
User avatar

Joined: Sun Oct 01, 2006 9:26 pm
Posts: 3768
Title: All in a day's work.
-> Programmering

The only things I like about the UPS format is that it supports files of arbitrary size and reversing patches ("unpatching" or "undoing" a previously applied patch). Other than that, it's still just IPS in sheep's clothing.

Re: encryption - No. See: viewtopic.php?p=76913#p76913

Re: compression - Negligible. Compression is "nice to have" but unless you're dealing with bandwidth or storage space sensitivity, compression is just a means to add unnecessary complexity.


What's really needed in a patch format is the following:
  1. Simple spec
  2. Arbitrary file and patch size limits
  3. Binary deltas

Everything else (checksum/hash verification, compression, embedded descriptions, tags, credentials, etc.) is firmly in the "nice to have but not really important" category.

Some examples of currently available [open source] "binary delta patching" software:

To my knowledge, these all cover points 2 and 3 above. But point #1? Hmmm...

_________________
I have to return some video tapes.

Feed me a stray cat.


Top
 Profile  
Reply with quote  
 Post subject: Re: Patching Format Idea
PostPosted: Wed Apr 28, 2010 9:49 pm 
Offline
Komrade
Komrade

Joined: Tue Mar 27, 2007 10:18 am
Posts: 1328
Any format I come up with would at least need extendability if not including the features I want because I have different needs.

_________________
Image


Top
 Profile  
Reply with quote  
 Post subject: Re: Patching Format Idea
PostPosted: Tue Jul 27, 2010 7:42 pm 
Offline
Komrade
Komrade

Joined: Tue Mar 27, 2007 10:18 am
Posts: 1328
I went ahead and did this.

Hextator's Doc/$Development/@Misc/New Patching Format

Personally I think it's glorious, but whatever.

I hated the VLI format you were thinking about for the SRDP and went and made my own. It's moderately efficient and definitely simple, but nothing revolutionary.

_________________
Image


Top
 Profile  
Reply with quote  
 Post subject: Re: Patching Format Idea
PostPosted: Tue Jul 27, 2010 10:26 pm 
Offline
Kommunist
Kommunist
User avatar

Joined: Sat Nov 24, 2007 6:04 pm
Posts: 647
Location: Rattlesnakes and Sand
Zeld wrote:
Personally I think it's glorious, but whatever.


Or so you keep telling me over AIM continuously since you started work on this all of a day or so ago?


Top
 Profile  
Reply with quote  
 Post subject: Re: Patching Format Idea
PostPosted: Wed Jul 28, 2010 7:20 am 
Offline
Krew (Admin)
Krew (Admin)
User avatar

Joined: Sun Oct 01, 2006 9:26 pm
Posts: 3768
Title: All in a day's work.
I don't see anything about delta patching?

Until it can create/apply a minimal patch against an original and modified file system*, it's no better than IPS/UPS.


Also, an alternative to VLI is arbitrary precision numbers. For example, these functions in GMP would be perfect for getting/setting numbers of unknown size: http://gmplib.org/manual/Integer-Import-and-Export.html

The only problem I can think of is a portability issue: size_t (for the count variable) is the native word size on the processor; 32-bit on 32-bit CPUs, 64-bit on 64-bit CPUs. To make it truly portable, you would have to stick to the least common denominator, which, according to the C99 spec, is 16-bits: http://en.wikipedia.org/wiki/Size_t

Just use a size of 1, and you have a binary format like this: <uint16_t count>,<uint8_t array[count]>. "Sadly" that limits you to a number between 8 bits and 524,280 bits.


* The "file system" may be a disk image, ISO, NDS ROM, whatever; the point is, once a file in the system grows or shrinks (and the file system is "defragmented" or "rebuilt"), file data moves. A simple byte-by-byte compare of such a modification will typically lead to a patch that contains most (if not all) of the modified file, rather than a minimal series of patches (changes, additions, subtractions, deltas/moved data, etc.)

_________________
I have to return some video tapes.

Feed me a stray cat.


Top
 Profile  
Reply with quote  
 Post subject: Re: Patching Format Idea
PostPosted: Wed Jul 28, 2010 3:47 pm 
Offline
Komrade
Komrade

Joined: Tue Mar 27, 2007 10:18 am
Posts: 1328
It needs revision anyway. I had in the back of my head this eerie feeling that storing offsets was "too easy" and then I was reminded by the UPS spec (which I finally found) that I should be storing relative offsets instead of absolute ones to truly take advantage of the VLIs.

Yes, this is only a minor improvement over UPS at a glance, until you realize just how annoying the UPS format's integrity specification is to use in practice.

The spec and poc have been updated to use relative offsets and the source code for the poc now has more comments.

_________________
Image


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC - 8 hours [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group