Subject: [IGS-DCWG-55] RE: [IGS-DCWG-53] Fwd: Feedback: ZIMM station - daily data unavailability
From: "Schmidt, Michael" <[email-masked]>
To: "Carey Noll" <[email-masked]>, <[email-masked]>
Cc: <[email-masked]>
Date: Fri, 19 Mar 2010 15:18:18 -0400

******************************************************************************
IGS-DCWG Mail      19 Mar 12:54:13 PDT 2010      Message Number 55
******************************************************************************

Author: Michael Schmidt

Hi Carey

Thanks for raising this - this is a good time to be discussing the
robustness of data distribution / availability so that we can perhaps
arrive at some firm proposals in Newcastle.

So to follow up from our conversation and in response to the emails
below and from my reading of the position paper it would appear that the
main issues are:

1) Robust distribution of data from Operational Data Centers (ODC) to
Global Data Centers (GDC);
2) Robust updating of data when resubmissions are necessary;
3) synchronicity between GDC's
4) notification of resubmissions

In the context of this discussion, the term ODC would include all those
agencies / sites currently pushing data from one or more sites to one or
more GDC's.

If I understand you and others correctly, there is currently no
mirroring or synchronization between GDC's. As we all know this can lead
to one or a combination of the following:
	- incomplete data holdings;
	- incomplete / incorrect data holdings when taking
	  into account data resubmissions;

Addressing items 1-3:
=====================
To start to resolve this can we consider the following:

(A) ODC's push data to all GDC's;
(B) ODC's push resubmissions to all GDC's;

- A+B would/should address issues 1, 2 and 3 above except in those
instance when the availability of or communication to a GDC is
compromised, in which case there may be a latency.
- A simple system of book keeping at each GDC that could be interrogated
electronically might be desirable (ties into the notification issue).

I recognize that not all ODC's can necessarily meet this suggestion, but
would hazard the guess that most could (TBC).

Addressing item 4:
==================
(notification of re-submissions)
Currently this is handled by issuing Advisories through the IGSSTATION
list. It is not clear to me that this is the most effective method for a
number of reasons:
I) latency in receiving information;
   -in other words by the time the email is sent and
    acted upon by the user(s)it may be too late to be
    effective;
   - from the perspective of the IGS the primary users
     are the Analysis Centers but at the same time
     recognizing that there is a large user group
     outside the IGS.
II) incompleteness; I'll be the first to admit that we have not always
been consistent in sending out these notices; I suspect we are not
alone.
- other?

So perhaps the best place to start is by defining / examining:
- who needs notifications?
- design latency criteria of notifications;
- determine best method(s) for distribution;
- consider electronic book keeping method that can be interrogated by
automated processes;

It would seem that as we become more and more reliant on automated
processes, that automatic detection of resubmissions would be desirable.
As a start one could envision automated emails, as part of the
resubmission process by the ODC to a specific distribution list
consisting of the primary users of IGS data files. These emails could,
for example, be in a specific format, thus machine readable and
actionable (??).

At any rate I think this is a good discussion to be having now and
hopefully moving forward even in incremental steps.

Thanks again

Cheers

Mike

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] On Behalf Of Carey Noll
Sent: Tuesday, March 16, 2010 7:25 AM
To: [email-masked]
Cc: [email-masked]; Carey Noll
Subject: [IGS-DCWG-53] Fwd: Feedback: ZIMM station - daily data
unavailability

************************************************************************
******
IGS-DCWG Mail      16 Mar 07:23:50 PDT 2010      Message Number 53
************************************************************************
******

Author: Carey Noll/CDDIS

     Our group has not been very active in the last year or so.
I would like to begin discussions again on how we can best
serve our user communities and the IGS by ensuring that data
continue to flow through various channels despite outages of
a data center, at any level in the hierarchy (global, regional,
or operational). We have had several instances recently where
a data center was down and some data were not made available
because of the outage. Also, there are instances as shown
below where files are delivered but not propagated through the
data center chain.

     If you recall, we had a detailed plan presented during
the Darmstadt workshop:
ftp://igscb.jpl.nasa.gov/pub/resource/pubs/06_darmstadt/IGS%20WS%202006%
20Papers%20PDF/5_Bruyninx_posppr_proceedings_final.pdf
Following the workshop, efforts to generate an implementation
plan did not get off the ground. I would like to ask the group
to revisit this plan and come up with ideas on how to implement
the proposed steps, or a subset of them, in order to provide a
more redundant, robust data flow. We can entertain new ideas as
well. It is important for the data centers, working with the IGS
Infrastructure Committee, to address this issue.

     My goal is to have clear steps in mind prior to July's IGS
Workshop; there will be a data center/data flow presentation that
would be the best place to outline concrete steps for solving our
problems. I have also proposed that we hold a DCWG splinter
meeting at the workshop. Any suggestions on other topics to be
addressed during that meeting are welcome.

Regards,
Carey.
-----
Begin forwarded message:

> From: Robert Khachikyan <[email-masked]>
> Date: March 15, 2010 1:17:01 PM EDT
> To: G|nter Stangl <[email-masked]>
> Cc: Brockmann Elmar LT <[email-masked]>,
> "[email-masked]'" <[email-masked]>, IGS Central Bureau
<[email-masked]
> >, Habrich Heinz <[email-masked]>, "Noll, Carey E.
> (GSFC-6901)" <[email-masked]>
> Subject: Re: Feedback: ZIMM station - daily data unavailability
>
> Just an FYI, there might be a discussion going on with the IGS Data
> Centers to have all IGS data sync'ed. Carey might have additional
> information.
>
> Best Regards,
> --Robert Khachikyan
> IGS Central Bureau
>
> On 3/15/2010 6:02 AM, G|nter Stangl wrote:
>> Brockmann Elmar LT schrieb:
>>
>>> Dear Carine, Dear Dominique, Dear Guenter,
>>>
>>> We got the mail below, indicating that the daily data of ZIMM are
>>> not available at DOY 067, 071, 073 at data center OLG.
>>>
>>> I checked the upload protocol and saw, that the uploads were
>>> successfull to 6 DC (see below). I assume, that some internal
>>> problems at OLG are responsible for the missing data of ZIMM. By
>>> the way - the data of twin station ZIM2 are listed as available.
>>> There were also quite some "gaps" in the OLG list, so probably
>>> many station managers got a similar mail.
>>>
>>> You suggested, to "Please upload the missing data or, if
>>> applicable, send a EUREF mail to notify the EUREF community of the
>>> origin of the
>>> data gap."
>>> We do a "re-sent" in an automated way (5 trails very full hour).
>>> If data could not be delivered within 5 hours, we don't do a re-
>>> sending of data.
>>>
>>> I'm a little confused that you asked for re-uploading data. If the
>>> data are missing for any reason at one of the two European data
>>> centers (BKG, OLG) and in case of ZIMM even at the global DCs SIO,
>>> CDDIS, and IGN, I would expect, that the data centers will
>>> syncronisze the data. That's the reason for giving a personal
>>> feedback to the probably automatically generated mail.
>>>
>>> Greetings from Berne,
>>> Elmar
>>>
>>>
>>>
>>> Elmar Brockmann
>>> swisstopo
>>> Federal Office of Topography
>>> Geodesy
>>> Seftigenstrasse 264
>>> CH-3084 Wabern
>>> Phone   +41 31 963 21 11        operator
>>>         +41 31 963 22 56        direct
>>> Fax     +41 31 963 24 59
>>> [email-masked]
>>> www.swisstopo.ch
>>>
>>>
>>>
>>> ----------
>>>
>>>
>>> 067:
>>> zimm0670.10d.Z : was put to BKG         at 09-MAR-2010 00:04:11
>>> with size   335377
>>> zimm0670.10n.Z : was put to BKG         at 09-MAR-2010 00:04:17
>>> with size    33849
>>> zimm0670.10m.Z : was put to BKG         at 09-MAR-2010 00:04:17
>>> with size      893
>>> zimm0670.10d.Z : was put to SIO         at 09-MAR-2010 00:05:09
>>> with size   335377
>>> zimm0670.10n.Z : was put to SIO         at 09-MAR-2010 00:05:16
>>> with size    33849
>>> zimm0670.10m.Z : was put to SIO         at 09-MAR-2010 00:05:18
>>> with size      893
>>> zimm0670.10d.Z : was put to OLG         at 09-MAR-2010 00:06:17
>>> with size   335377
>>> zimm0670.10n.Z : was put to OLG         at 09-MAR-2010 00:06:24
>>> with size    33849
>>> zimm0670.10m.Z : was put to OLG         at 09-MAR-2010 00:06:25
>>> with size      893
>>> zimm0670.10d.Z : was put to AIUB        at 09-MAR-2010 00:07:59
>>> with size   335377
>>> zimm0670.10n.Z : was put to AIUB        at 09-MAR-2010 00:08:09
>>> with size    33849
>>> zimm0670.10m.Z : was put to AIUB        at 09-MAR-2010 00:08:10
>>> with size      893
>>> zimm0670.10d.Z : was put to CDDIS       at 09-MAR-2010 00:09:34
>>> with size   335377
>>> zimm0670.10n.Z : was put to CDDIS       at 09-MAR-2010 00:09:41
>>> with size    33849
>>> zimm0670.10m.Z : was put to CDDIS       at 09-MAR-2010 00:09:42
>>> with size      893
>>> zimm0670.10d.Z : was put to IGN         at 09-MAR-2010 00:10:41
>>> with size   335377
>>> zimm0670.10n.Z : was put to IGN         at 09-MAR-2010 00:10:49
>>> with size    33849
>>> zimm0670.10m.Z : was put to IGN         at 09-MAR-2010 00:10:50
>>> with size      893
>>>
>>>
>>> 071:
>>> zimm0710.10d.Z : was put to BKG         at 13-MAR-2010 00:04:05
>>> with size   334629
>>> zimm0710.10n.Z : was put to BKG         at 13-MAR-2010 00:04:12
>>> with size    34818
>>> zimm0710.10m.Z : was put to BKG         at 13-MAR-2010 00:04:14
>>> with size      859
>>> zimm0710.10d.Z : was put to SIO         at 13-MAR-2010 00:05:08
>>> with size   334629
>>> zimm0710.10n.Z : was put to SIO         at 13-MAR-2010 00:05:15
>>> with size    34818
>>> zimm0710.10m.Z : was put to SIO         at 13-MAR-2010 00:05:16
>>> with size      859
>>> zimm0710.10d.Z : was put to OLG         at 13-MAR-2010 00:06:15
>>> with size   334629
>>> zimm0710.10n.Z : was put to OLG         at 13-MAR-2010 00:06:21
>>> with size    34818
>>> zimm0710.10m.Z : was put to OLG         at 13-MAR-2010 00:06:22
>>> with size      859
>>> zimm0710.10d.Z : was put to AIUB        at 13-MAR-2010 00:07:51
>>> with size   334629
>>> zimm0710.10n.Z : was put to AIUB        at 13-MAR-2010 00:08:03
>>> with size    34818
>>> zimm0710.10m.Z : was put to AIUB        at 13-MAR-2010 00:08:04
>>> with size      859
>>> zimm0710.10d.Z : was put to CDDIS       at 13-MAR-2010 00:09:26
>>> with size   334629
>>> zimm0710.10n.Z : was put to CDDIS       at 13-MAR-2010 00:09:34
>>> with size    34818
>>> zimm0710.10m.Z : was put to CDDIS       at 13-MAR-2010 00:09:36
>>> with size      859
>>> zimm0710.10d.Z : was put to IGN         at 13-MAR-2010 00:10:41
>>> with size   334629
>>> zimm0710.10n.Z : was put to IGN         at 13-MAR-2010 00:10:50
>>> with size    34818
>>> zimm0710.10m.Z : was put to IGN         at 13-MAR-2010 00:10:52
>>> with size      859
>>>
>>> 073:
>>> zimm0730.10d.Z : was put to BKG         at 15-MAR-2010 00:04:05
>>> with size   334959
>>> zimm0730.10n.Z : was put to BKG         at 15-MAR-2010 00:04:11
>>> with size    34757
>>> zimm0730.10m.Z : was put to BKG         at 15-MAR-2010 00:04:14
>>> with size      876
>>> zimm0730.10d.Z : was put to SIO         at 15-MAR-2010 00:05:08
>>> with size   334959
>>> zimm0730.10n.Z : was put to SIO         at 15-MAR-2010 00:05:16
>>> with size    34757
>>> zimm0730.10m.Z : was put to SIO         at 15-MAR-2010 00:05:17
>>> with size      876
>>> zimm0730.10d.Z : was put to OLG         at 15-MAR-2010 00:06:16
>>> with size   334959
>>> zimm0730.10n.Z : was put to OLG         at 15-MAR-2010 00:06:23
>>> with size    34757
>>> zimm0730.10m.Z : was put to OLG         at 15-MAR-2010 00:06:24
>>> with size      876
>>> zimm0730.10d.Z : was put to AIUB        at 15-MAR-2010 00:08:01
>>> with size   334959
>>> zimm0730.10n.Z : was put to AIUB        at 15-MAR-2010 00:08:12
>>> with size    34757
>>> zimm0730.10m.Z : was put to AIUB        at 15-MAR-2010 00:08:13
>>> with size      876
>>> zimm0730.10d.Z : was put to CDDIS       at 15-MAR-2010 00:09:33
>>> with size   334959
>>> zimm0730.10n.Z : was put to CDDIS       at 15-MAR-2010 00:09:42
>>> with size    34757
>>> zimm0730.10m.Z : was put to CDDIS       at 15-MAR-2010 00:09:43
>>> with size      876
>>> zimm0730.10d.Z : was put to IGN         at 15-MAR-2010 00:10:42
>>> with size   334959
>>> zimm0730.10n.Z : was put to IGN         at 15-MAR-2010 00:10:51
>>> with size    34757
>>> zimm0730.10m.Z : was put to IGN         at 15-MAR-2010 00:10:52
>>> with size      876
>>>
>>>
>>>
>>>
>>>
>>>> -----Urspr|ngliche Nachricht-----
>>>> Von: [email-masked] [mailto:[email-masked]]
>>>> Gesendet: Montag, 15. Mdrz 2010 09:09
>>>> An: [email-masked]; Brockmann Elmar LT; Wild Urs LT;
>>>> [email-masked]
>>>> Betreff: ZIMM station - daily data unavailability
>>>>
>>>>
>>>> Dear colleagues,
>>>>
>>>> All daily data of the EPN stations should be uploaded to both
>>>> the OLG and BKG regional data centres.
>>>>
>>>> We noticed that from 067/2010 to 073/2010, the daily data of
>>>> the station ZIMM are incomplete. Missing are: at OLG: DOY
>>>> 067/2010, 071/2010, 073/2010
>>>>
>>>> Please upload the missing data or, if applicable, send a
>>>> EUREF mail to notify the EUREF community of the origin of the
>>>> data gap. Consult
>>>> http://www.epncb.oma.be/ftp/center/data/BKGE.RDC and
>>>> http://www.epncb.oma.be/ftp/center/data/OLG.RDC for data
>>>> upload instructions.
>>>>
>>>> Detailed daily data holding information is available from BKG and
>>>> OLG:
>>>> BKG: ftp://igs.bkg.bund.de/EUREF/obs/2010/CHECK_IMPORT.BKG or ...
>>>>      ftp://igs.bkg.bund.de/IGS/obs/2010/CHECK_IMPORT.BKG
>>>> OLG: ftp://olggps.oeaw.ac.at/pub/outdata/check_import.OLG
>>>>
>>>> Best regards,
>>>>
>>>> EPN Central Bureau / Carine and Dominique
>>>> [email-masked]
>>>>
>>>>
>>>>
>>>
>>>
>> Dear colleagues,
>>
>> as you can see from the attachment the daily files of ZIMM were
>> corrupt
>> at DOY 067, 071 and 073 (obs files only, not nav and met files).
>>
>> Concerning the general statement of Elmar, synchronization between
>> BKG
>> and OLG was done until some years ago, but stopped because BKG
>> complained a too heavy loading of requests (OLG used wget). We can
>> discuss this again, perhaps using a time window of synchronization
>> (three months?).
>>
>> kind regards
>>
>> Guenter
>>

-----
Ms. Carey Noll
Manager, Crustal Dynamics Data Information System (CDDIS)
Secretary, ILRS Central Bureau
NASA GSFC
Code 690.1
Greenbelt, MD 20771
USA

E-mail:  [email-masked]
Voice: (301) 614-6542
Fax: (301) 614-6015
WWW:  http://cddis.gsfc.nasa.gov