Subject: [IGS-DCWG-34] RE: [IGS-DCWG-29] Re: [IGS-DCWG-27] Any DCs accepting .gz RINEX?
From: "Schmidt, Michael" <[email-masked]>
To: 'Carey Noll' <[email-masked]>, "Angelyn W. Moore" <[email-masked]>
Cc: [email-masked]
Date: Thu, 18 Nov 2004 15:01:14 -0800

******************************************************************************
IGS-DCWG Mail      18 Nov 15:11:33 PST 2004      Message Number 34
******************************************************************************

Author: Michael Schmidt

Hi everybody

To partially answer Carey's question of .Z vs. .gz vs. .zip files, here is a
brief comparison of compression of a 1 hr, 1Hz file (on Solaris):

-rw-r--r--   1 wcdaop   gpsusers  933624 Nov 18 21:49 bcdl320a.04d
-rw-r--r--   1 wcdaop   gpsusers  297759 Nov 18 21:49 bcdl320a.04d.gz
-rw-r--r--   1 wcdaop   gpsusers  291365 Nov 18 21:48 bcdl320a.04d.Z
-rw-r--r--   1 wcdaop   gpsusers  297884 Nov 18 22:11 bcdl320a.04d.zip

-rw-r--r--   1 wcdaop   gpsusers 2583764 Nov 17 17:55 bcdl320a.04o
-rw-r--r--   1 wcdaop   gpsusers  898794 Nov 17 17:55 bcdl320a.04o.gz
-rw-r--r--   1 wcdaop   gpsusers  961869 Nov 17 17:55 bcdl320a.04o.Z
-rw-r--r--   1 wcdaop   gpsusers  898919 Nov 18 22:10 bcdl320a.04o.zip

Note that the .gz and .zip compressions of the .04o file is slightly better
than the .Z compression but that the opposite is true for the .04d file.

Wrt which one to use, my 2cents(Cdn) worth:

a) it would not appear that there is a huge space saving of one over the
other {from this limited, though highly scientific test ... ahem .. :>) }

b) it would be appropriate to examine which format is best for our user
base, specifically the none-IGS user base, and that would include the use of
the Hatanaka compression (I can't tell you how many phone calls / emails I
have had on that one ...); for the purpose of this discussion I am assuming
that most core IGS data / analysis centres can handle any of the above,
including Hatanaka. However the same cannot necessarily be said for the
non-IGS users:
- commercial programs can uncompress any of the above (except Hatanaka) but
cannot necesssarily re-compress them in that format (based on examining
Winzip's web page).
- what format can commercial GPS packages handle? In other words how can we
make it easier for the non-expert to handle IGS data streams and thus
encourage / increase use of our products?
- is it reasonable to continue with the Hatanaka compression given that
there are few commercial applications that can handle it? - I believe some
of the vendor software packages can but is this universal?
- should keep in mind that many people do not know how to use DOS programs
(including the latest batch of grad students we've had) so in the B.Gates
world, utilities really have to be Windows based; I believe the gzip
utilities are only available for DOS, not Windows (???), similarly Hatanaka
(except as noted above).

c) my suggestion would be the following:
i) status quo for the time being;
ii) do a little bit of research to see what the optimal compression scheme
would be and what tools are available from  / supported by the GNSS vendors
and at that point make any adjustments as required.
iii) roll out the change(s) once we are sure that these will address current
/ future needs.
iv) don't bother converting old data files, but leave that up to individual
data centres.

I tend to agree with Angie that .Z may become a "legacy" format and that the
gzip format will probably become more widely used. I would be curious to
hear what the common compression formats outside of N. America are,
especially if different from the above.

end of 2 cents - hope I have not overcharged :>)

cheers

Mike

PS if we had a compression scheme as effective as the proprietaty Trimble
T00 format all of this would be a mute point ....

----------------------------------------------------------------------------
Michael Schmidt, P.Eng.               [email-masked]
TEL +1 250-363-6760    FAX +1 250-363-6565  Mobile +1 250-920-6128
Geological Survey of Canada           Commission géologique du Canada
Natural Resources Canada              Ressources naturelles Canada
PO Box 6000                           B.P. 6000
Sidney BC Canada V8L 4B2              Sidney (C.B.) Canada V8L 4B2
Street:                               Rue:
  9860 West Saanich Rd                  9860 West Saanich Rd
  North Saanich BC                      North Saanich (C.B.)
  Canada V8L 3S1                        Canada V8L 3S1

Government of Canada                  Gouvernement du Canada

                         www.pgc.nrcan.gc.ca
----------------------------------------------------------------------------

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Carey Noll
> Sent: Thursday, November 18, 2004 1:10 PM
> To: Angelyn W. Moore
> Cc: [email-masked]
> Subject: [IGS-DCWG-29] Re: [IGS-DCWG-27] Any DCs accepting .gz RINEX?
> 
> 
> **************************************************************
> ****************
> IGS-DCWG Mail      18 Nov 13:15:21 PST 2004      Message Number 29
> **************************************************************
> ****************
> 
> Author: Carey Noll
> 
> Angie,
> 
>    CDDIS can certainly do a translation of a file from .gz to 
> Z as long as we know who is doing it ahead of time so the 
> scripts can be modified of course.  There may also be some 
> delay in data availability by adding this extra step of 
> course (minor).
> 
>    A large archive like CDDIS does not really want to re-do 
> the entire archive in another compression scheme!  Especially 
> since we are trying to get the archive up and running on a 
> new server.  I don't see the benefit of doing this, but I am 
> probably wearing blinders.
> 
> Carey.
> 
> +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
> +=+=+=+=+=
> ++
> Ms. Carey Noll  Manager, Crustal Dynamics Data Information 
> System (CDDIS)
>                                            Secretary, ILRS 
> Central Bureau
> Code 922                                     E-mail:  
> [email-masked]
> NASA GSFC                                           Voice: 
> (301) 614-6542
> Greenbelt, MD 20771                                   Fax: 
> (301) 614-6099
> USA                  WWW:  
> http://cddisa.gsfc.nasa.gov/cddis_welcome.html
> 
> +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
> +=+=+=+=+=
> ++
>