Export to GitHub

ziparchive - issue #22

Adding large filles problem


Posted on Apr 21, 2011 by Massive Kangaroo
  1. On some devices (iPad in my case) trying to add a large file in -(BOOL) addFileToZip:(NSString*) file newname:(NSString*) newname; will crash the app because the method [NSData dataWithContentsOfFile:] will use up all the memory.

I fixed the issue for my particular problem with zip files that have no password by changing the addFileToZip method a bit. You might want to take a look at it and change it universally... Also there is no internal autorelease pool needed when adding multiple files.

The methode:

define M_FRAGMENT_SIZE 10000000

-(BOOL) addFileToZip:(NSString*) file newname:(NSString*) newname; { if( !_zipFile ) return NO;

// tm_zip filetime; time_t current; time( &current );

zip_fileinfo zipInfo = {0};

// zipInfo.dosDate = (unsigned long) current;

NSDictionary* attr = [[NSFileManager defaultManager] attributesOfItemAtPath:file error:nil];
if( attr )
{
    NSDate* fileDate = (NSDate*)[attr objectForKey:NSFileModificationDate];
    if( fileDate )
    {
        // some application does use dosDate, but tmz_date instead
    //  zipInfo.dosDate = [fileDate timeIntervalSinceDate:[self Date1980] ];
        NSCalendar* currCalendar = [NSCalendar currentCalendar];
        uint flags = NSYearCalendarUnit | NSMonthCalendarUnit | NSDayCalendarUnit | 
            NSHourCalendarUnit | NSMinuteCalendarUnit | NSSecondCalendarUnit ;
        NSDateComponents* dc = [currCalendar components:flags fromDate:fileDate];
        zipInfo.tmz_date.tm_sec = [dc second];
        zipInfo.tmz_date.tm_min = [dc minute];
        zipInfo.tmz_date.tm_hour = [dc hour];
        zipInfo.tmz_date.tm_mday = [dc day];
        zipInfo.tmz_date.tm_mon = [dc month] - 1;
        zipInfo.tmz_date.tm_year = [dc year];
    }
}

int ret ;
NSData* data = nil;
if( [_password length] == 0 )
{
    ret = zipOpenNewFileInZip( _zipFile,
                              (const char*) [newname UTF8String],
                              &zipInfo,
                              NULL,0,
                              NULL,0,
                              NULL,//comment
                              Z_DEFLATED,
                              Z_DEFAULT_COMPRESSION );
}
else
{
    FILE *f = fopen([file cStringUsingEncoding:NSUTF8StringEncoding], "r");
    fseek(f, 0, SEEK_END);
    long fLenght = ftell(f);
    void *fBuffer = malloc(fLenght);
    fread(fBuffer, 1, fLenght, f);
    fclose(f);
    data = [[NSData alloc] initWithBytesNoCopy:fBuffer length:fLenght];
    uLong crcValue = crc32( 0L,NULL, 0L );
    crcValue = crc32( crcValue, (const Bytef*)[data bytes], [data length] );
    ret = zipOpenNewFileInZip3( _zipFile,
                              (const char*) [newname UTF8String],
                              &zipInfo,
                              NULL,0,
                              NULL,0,
                              NULL,//comment
                              Z_DEFLATED,
                              Z_DEFAULT_COMPRESSION,
                              0,
                              15,
                              8,
                              Z_DEFAULT_STRATEGY,
                              [_password cStringUsingEncoding:NSASCIIStringEncoding],
                              crcValue );
}
if( ret!=Z_OK )
{
    [data release];
    return NO;
}

// M_FRAGMENT_SIZE 10000000 (10MB) FILE *f = fopen([file cStringUsingEncoding:NSUTF8StringEncoding], "r"); if(!f) return NO;

fseek(f, 0, SEEK_END);
long fLenght = ftell(f);
rewind(f);
void *fBuffer = malloc(M_FRAGMENT_SIZE);

for (;fLenght > M_FRAGMENT_SIZE; fLenght-=M_FRAGMENT_SIZE) {        
    fread(fBuffer, 1, M_FRAGMENT_SIZE, f);
    ret = zipWriteInFileInZip( _zipFile, (const void*)fBuffer, M_FRAGMENT_SIZE);
}   
if(fLenght) {
    fread(fBuffer, 1, fLenght, f);
    ret = zipWriteInFileInZip( _zipFile, (const void*)fBuffer, fLenght);
    if( ret!=Z_OK )
    {
        free(fBuffer);
        fclose(f);
        return NO;
    }
}
free(fBuffer);
fclose(f);

if( ret!=Z_OK )
{
    return NO;
}
ret = zipCloseFileInZip( _zipFile );
if( ret!=Z_OK )
    return NO;
return YES;

}

Now the memory consumption is restricted by M_FRAGMENT_SIZE in bytes, currently 10MB and it seems to work great...

Comment #1

Posted on May 19, 2011 by Swift Cat

You could have just changed to using dataWithContentsOfMappedFile: which will mmap the file into memory rather than mallocing anything and so deal with large files that way.

Comment #2

Posted on Apr 6, 2015 by Swift Bear

i have tried you could and was able to send 279 mb file on web server.i can say it is working.I have added @autorelease pool in for{} loop for more frequent memory release

Status: New

Labels:
Type-Defect Priority-Medium