Перейти в статусOPMET compareVAA graphics

Service Monitoring 2026-06-08 12:59:06 UTC - 3h25m22s

Now: 2026-06-08 16:24:29 UTCAction trace:enabledTrace file:/app/tmp/service_actions.log
PostgreSQL
Status:connected
Config:DB_URL / DATABASE_URL
Value:postgres://weather:***@new_weather_db/weather
Connection:postgres://new_weather_db/weather
Check: OK (0 ms)
METAR decoder
Status:connected
Config:METAR_TAF_DECODE_URL / METS_DECODER_URL
Value:http://metar_taf_parser:7000
Health URL:http://metar_taf_parser:7000/healthcheck
HTTP: 200 (6 ms)
OPMET parser (primary RB)
Status:connected
Config:ASG_MET_PARSER_PRIMARY / *_URL
Value:http://opmet_parser:7000
Health URL:http://opmet_parser:7000/healthcheck
HTTP: 200 (12 ms)
OPMET parser (shadow CS)
Status:connected
Config:ASG_MET_PARSER_SHADOW / *_URL
Value:http://old_meteo_text_parser:5001
Health URL:http://old_meteo_text_parser:5001/
HTTP: 200 (4 ms)
TAF decoder
Status:connected
Config:METAR_TAF_DECODE_URL / METS_DECODER_URL
Value:http://metar_taf_parser:7000
Health URL:http://metar_taf_parser:7000/healthcheck
HTTP: 200 (6 ms)

Потоки данных: что собираем и куда пишем

PipelineSourceCollected dataDestination tables
NOAA METARNOAA metars.cache.csv.gzMETAR/SPECI: ICAO, observed_at, raw_text, weather, wind, pressure, cloudsnoaa_metars noaa_cache_runs noaa_ingest_state aerodromes
NOAA TAFNOAA tafs.cache.xml.gzTAF: ICAO, issued/valid range, raw_text, decoded forecast groupsnoaa_tafs noaa_cache_runs noaa_ingest_state
ArcGIS Tropical CyclonesArcGIS Active_Hurricanes_v1 FeatureServerObserved/forecast positions, tracks, cones, warnings, wind swaths and radii polygonsactive_tropical_cyclones
SADIS Tropical Cyclone AdvisoriesSADIS FTP /TROPICAL_CYCLONE_ADVISORIESWMO/TAC tropical cyclone outlooks and advisories as raw bulletinstropical_cyclone_raw_payloads
ArcGIS Recent Tropical CyclonesArcGIS Recent_Hurricanes_v1 FeatureServerObserved positions, observed tracks, observed wind swaths for recent systemsrecent_tropical_cyclones
Roshydro ingestFTP *.b files (ROSHYDRO_FTP_*)METAR/TAF/OPMET batches + parser outputs + file processing statsroshydro_metars roshydro_tafs roshydro_opmet_processed_files aerodromes opmetdata sigmet_geo airmet_geo gamet_geo
Roshydro OPMET parity (code present, runtime switch pending)Same FTP *.b, ETX/=//END PART splittingAIRMET/SIGMET/GAMET normalized records, parse/build errors, geo projectionsopmetdata opmeterror sigmet_geo airmet_geo gamet_geo
Volcanic Ash AdvisorySADIS FTP + configured VAAC web sourcesVAA raw text, VAAC, volcano, issue/valid times, ash cloud altitude, movement and geometryvolcanic_ash_sources volcanic_ash_advisories volcanic_ash_clouds volcanic_ash_graphics

Active Tropical Cyclones

Existsyes
Rows111
Distinct storms5
Latest fetch2026-06-08 16:14:08 UTC
Ago10m21s

Sources

Source serviceRowsStormsLatest fetchAgo
ARCGIS_ACTIVE_HURRICANES_V111042026-06-08 16:14:08 UTC10m21s
JTWC_SIGNIFICANT_TROPICAL_WEATHER_ADVISORY112026-05-26 03:03:29 UTC13d13h21m

Source layers

Source serviceLayer idLayerRowsLatest fetchAgo
ARCGIS_ACTIVE_HURRICANES_V10Forecast Position172026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V11Observed Position502026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V12Forecast Track32026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V13Observed Track92026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V14Forecast Error Cone and Danger Area32026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V15Watches and Warnings22026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V17Tropical Storm Force (34kts)102026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V110Raw 1/10th Degree Data (All)152026-06-08 16:14:08 UTC10m21s
ARCGIS_ACTIVE_HURRICANES_V111Observed Wind Swath12026-06-08 16:14:08 UTC10m21s
JTWC_SIGNIFICANT_TROPICAL_WEATHER_ADVISORY3001Tropical Disturbance Summary12026-05-26 03:03:29 UTC13d13h21m

Recent Tropical Cyclones

Existsyes
Rows2988
Distinct storms48
Latest fetch2026-06-08 13:09:08 UTC
Ago3h15m20s

Sources

Source serviceRowsStormsLatest fetchAgo
ARCGIS_RECENT_HURRICANES_V11125282026-06-05 03:00:00 UTC3d13h24m29s
JTWC_RECENT_TROPICAL_CYCLONES_BDECK_UNOFFICIAL_MIRROR655382026-06-08 13:09:08 UTC3h15m20s
NOAA_IBTRACS_RECENT_CSV1208312026-06-08 01:07:50 UTC15h16m38s

Source layers

Source serviceLayer idLayerRowsLatest fetchAgo
ARCGIS_RECENT_HURRICANES_V10Observed Position9412026-06-05 03:00:00 UTC3d13h24m29s
ARCGIS_RECENT_HURRICANES_V11Observed Track1842026-06-05 03:00:00 UTC3d13h24m29s
JTWC_RECENT_TROPICAL_CYCLONES_BDECK_UNOFFICIAL_MIRROR2100Observed Position6092026-06-08 13:09:08 UTC3h15m20s
JTWC_RECENT_TROPICAL_CYCLONES_BDECK_UNOFFICIAL_MIRROR2101Observed Track462026-06-08 01:12:48 UTC15h11m40s
NOAA_IBTRACS_RECENT_CSV2000Observed Position11792026-06-08 01:07:50 UTC15h16m38s
NOAA_IBTRACS_RECENT_CSV2001Observed Track292026-05-27 06:51:40 UTC12d9h32m48s

Tropical Cyclone Raw Payloads

Existsyes
Rows8042
Latest fetch2026-06-08 16:17:56 UTC
Ago6m32s

Sources

Source serviceRowsLatest fetchAgo
JTWC_SIGNIFICANT_TROPICAL_WEATHER_ADVISORY22026-05-26 03:03:29 UTC13d13h21m
NOAA_NHC_GIS_RSS79772026-06-08 16:17:56 UTC6m32s
SADIS_TROPICAL_CYCLONE_ADVISORIES632026-06-08 16:10:23 UTC14m6s
Products:nhc_gis_rss=7977,significant_tropical_weather_advisory=2,tropical_cyclone_advisory=63Statuses:no_active_cyclones=7571,raw=471

Recent payloads

SourceProductStatusBasinStormIssuedFetchedAgo
NOAA_NHC_GIS_RSScentral_pacificno_active_cyclonescentral_pacific——2026-06-08 16:17:56 UTC6m32s
NOAA_NHC_GIS_RSSeast_pacificraweast_pacific——2026-06-08 16:17:56 UTC6m32s
NOAA_NHC_GIS_RSSatlanticno_active_cyclonesatlantic——2026-06-08 16:17:56 UTC6m32s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ22_KNHC_081455raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ23_KNHC_081456raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ22_KNHC_080854raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ21_KNHC_080233raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ22_KNHC_080234raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ22_KNHC_072040raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ21_KNHC_072034raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ22_KNHC_071456raw———2026-06-08 16:10:23 UTC14m6s
SADIS_TROPICAL_CYCLONE_ADVISORIESFKPZ21_KNHC_071445raw———2026-06-08 16:10:23 UTC14m6s

Volcanic Ash Advisory

Pipeline:enabledLast run: 2026-06-08 16:24:13 UTCRun status: okAgo: 16sFreshness: warn 120 min / critical 360 min
Sources: 6Advisories: 2202Clouds: 2822Graphics: 158 (georeferenced=73,no_geometry=11,parsed=74)State: fresh=3 warning=2 critical=1 missing=0
SourceVAACStatusLatest issueAgoLatest advisory24h48hTotalLast error
DARWIN_BOM_AGGREGATEDARWINcritical——— / —21 adv / 76 clouds43 adv / 157 clouds255 adv / 909 clouds2026-06-08T16:16:38Z - Rufus::Scheduler::TimeoutError
MONTREAL_ECCCMONTREALfresh——— / —0 adv / 0 clouds0 adv / 0 clouds1228 adv / 0 clouds—
SADIS_FTPMULTIfresh2026-06-08 15:00:00 UTC1h24m29sSHEVELUCH / 2026/19039 adv / 133 clouds77 adv / 257 clouds497 adv / 1388 clouds—
TOKYO_JMATOKYOfresh2026-06-08 15:00:00 UTC1h24m29sSHEVELUCH / 2026/1908 adv / 35 clouds23 adv / 79 clouds125 adv / 214 clouds—
TOULOUSE_METEO_FRTOULOUSEwarning2026-06-08 14:00:00 UTC2h24m29sPITON DE LA FOURNAISE / 2026/11 adv / 0 clouds1 adv / 0 clouds11 adv / 29 clouds—
WASHINGTON_OSPOWASHINGTONwarning2026-06-08 13:32:00 UTC2h52m29sFUEGO / 2026/6664 adv / 13 clouds8 adv / 29 clouds86 adv / 282 clouds—

Таблицы и последние изменения данных

TableExistsRowsLatestAgoError
aerodromesyes56492026-06-05 23:23:51 UTC2d17h37s—
noaa_metarsyes1935862026-06-08 16:20:00 UTC4m29s—
noaa_tafsyes294952026-06-08 16:21:00 UTC3m29s—
roshydro_metarsyes1070822026-06-08 16:20:00 UTC4m29s—
roshydro_tafsyes280542026-06-08 17:00:00 UTCnow—
active_tropical_cyclonesyes1112026-06-08 16:14:08 UTC10m21s—
recent_tropical_cyclonesyes29882026-06-08 13:09:08 UTC3h15m20s—
tropical_cyclone_raw_payloadsyes80422026-06-08 16:17:56 UTC6m32s—
roshydro_opmet_processed_filesyes3451942026-06-08 16:24:26 UTC3s—
noaa_cache_runsyes1299152026-06-08 16:24:25 UTC3s—
noaa_ingest_stateyes2———
opmetdatayes287362026-06-08 16:24:26 UTC3s—
opmeterroryes0———
sigmet_geoyes148332026-06-08 16:24:26 UTC3s—
airmet_geoyes31442026-06-08 16:10:20 UTC14m9s—
gamet_geoyes107592026-06-08 16:15:28 UTC9m—
volcanic_ash_sourcesyes62026-06-08 16:24:08 UTC20s—
volcanic_ash_advisoriesyes22022026-06-08 15:00:00 UTC1h24m29s—
volcanic_ash_cloudsyes28222026-06-09 08:20:00 UTCnow—
volcanic_ash_graphicsyes1582026-06-08 13:32:00 UTC2h52m29s—

Runtime env snapshot

VariableValue
RECENT_TROPICAL_CYCLONES_SOURCEALL
ACTIVE_TROPICAL_CYCLONES_SOURCEALL
TROPICAL_CYCLONES_JOB_INTERVAL_MIN15
RECENT_TROPICAL_CYCLONES_JOB_INTERVAL_MIN15
RECENT_TROPICAL_CYCLONES_LOOKBACK_DAYS130
ROSHYDRO_JOB_INTERVAL_MIN—
NOAA_METAR_JOB_INTERVAL_MIN—
NOAA_TAF_JOB_INTERVAL_MIN—
NOAA_METAR_CACHE_URL—
NOAA_TAF_CACHE_URL—
ARCGIS_TROPICAL_CYCLONES_URLhttps://services9.arcgis.com/RHVPKKiFTONKtxq3/ArcGIS/rest/services/Active_Hurricanes_v1/FeatureServer
ARCGIS_RECENT_TROPICAL_CYCLONES_URLhttps://services9.arcgis.com/RHVPKKiFTONKtxq3/ArcGIS/rest/services/Recent_Hurricanes_v1/FeatureServer
NOAA_IBTRACS_RECENT_CSV_URLhttps://www.ncei.noaa.gov/data/international-best-track-archive-for-climate-stewardship-ibtracs/v04r01/access/csv/ibtracs.last3years.list.v04r01.csv
JTWC_TROPICAL_CYCLONES_PRODUCTS_BASE_URLhttps://www.metoc.navy.mil/jtwc/products/
JTWC_RECENT_TROPICAL_CYCLONES_BDECK_UNOFFICIAL_MIRROR_BASE_URLhttps://hurricanes.ral.ucar.edu/repository/data/bdecks_open/
SADIS_FTP_TC_ADVISORIES_DIR/TROPICAL_CYCLONE_ADVISORIES
SADIS_TROPICAL_CYCLONES_FETCH_LIMIT50
VOLCANIC_ASH_PIPELINE_ENABLEDtrue
VOLCANIC_ASH_SOURCESSADIS,WASHINGTON,TOKYO,MONTREAL,TOULOUSE,DARWIN_AGGREGATE
VOLCANIC_ASH_FETCH_LIMIT50
VOLCANIC_ASH_JOB_INTERVAL_MIN15
VOLCANIC_ASH_FRESHNESS_WARN_MIN120
VOLCANIC_ASH_FRESHNESS_CRITICAL_MIN360
SADIS_FTP_HOSTsadisftp.metoffice.gov.uk
SADIS_FTP_VA_ADVISORIES_DIR/VOLCANIC_ASH_ADVISORY_STATEMENTS
ROSHYDRO_FTP_HOST193.7.166.1
ROSHYDRO_FTP_REMOTE_GLOB—
OPMET_COMPARE_ENABLEDfalse
OPMET_COMPARE_OUTPUT_DIRcompare_opmet
OPMET_COMPARE_LIMIT—
OPMET_ERROR_RETENTION_HOURS—
OPMET_ERROR_CLEANUP_INTERVAL_MIN—
METS_DECODER_URLhttp://metar_taf_parser:7000
ASG_MET_PARSER—
ASG_MET_PARSER_MODE—
ASG_MET_PARSER_PRIMARYRB
ASG_MET_PARSER_SHADOWCS
ASG_MET_PARSER_CS_URLhttp://old_meteo_text_parser:5001
ASG_MET_PARSER_RB_URLhttp://opmet_parser:7000
ACTION_TRACE_ENABLEDtrue

OPMET errors

Top error types:
created_atopmet_typeerror_typemessageopmetdata_idoriginal_file

Temporary action trace (TEST)

Это временный функционал для тестирования. После завершения тестов удалить:src/app/services/support/action_recorder.rb, маршрут/monitor/actions и вызовыActionRecorder.record(...)

[
  {
    "at": "2026-06-08T16:24:29Z",
    "source": "api.monitor",
    "status": "info",
    "action": "monitor_requested",
    "details": null
  },
  {
    "at": "2026-06-08T16:24:26Z",
    "source": "orchestrator.roshydro",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "files_count": 1
    }
  },
  {
    "at": "2026-06-08T16:24:26Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090923.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:24:25Z",
    "source": "orchestrator.noaa.taf",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 164,
      "skipped": 3003
    }
  },
  {
    "at": "2026-06-08T16:24:25Z",
    "source": "orchestrator.noaa.metar",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 231,
      "skipped": 4974
    }
  },
  {
    "at": "2026-06-08T16:24:23Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090923.b"
    }
  },
  {
    "at": "2026-06-08T16:24:23Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "files_detected",
    "details": {
      "count": 1
    }
  },
  {
    "at": "2026-06-08T16:24:13Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "sources": {
        "toulouse_meteo_fr": {
          "fetched": 1,
          "saved": 1,
          "errors": []
        }
      },
      "fetched": 1,
      "saved": 1,
      "errors": []
    }
  },
  {
    "at": "2026-06-08T16:24:08Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": {
      "source_names": [
        "TOULOUSE"
      ]
    }
  },
  {
    "at": "2026-06-08T16:23:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "dependencies_ok",
    "details": {
      "opmet_primary_mode": "RB",
      "opmet_primary_url": "http://opmet_parser:7000",
      "opmet_shadow_mode": "CS",
      "opmet_shadow_url": "http://old_meteo_text_parser:5001"
    }
  },
  {
    "at": "2026-06-08T16:23:08Z",
    "source": "orchestrator.noaa.taf",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:23:08Z",
    "source": "orchestrator.noaa.metar",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:23:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:22:21Z",
    "source": "orchestrator.roshydro",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "files_count": 10
    }
  },
  {
    "at": "2026-06-08T16:22:21Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090922.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090922.b"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090921.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "OMSJ",
      "observed_at": "2026-06-08T16:30:00Z",
      "now_utc": "2026-06-08T16:22:20Z"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "KTFP",
      "observed_at": "2026-06-08T16:55:00Z",
      "now_utc": "2026-06-08T16:22:20Z"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090921.b"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090920.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090920.b"
    }
  },
  {
    "at": "2026-06-08T16:22:20Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090919.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090919.b"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090918.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090918.b"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090917.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "BKPR",
      "observed_at": "2026-06-08T16:30:00Z",
      "now_utc": "2026-06-08T16:22:18Z"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090917.b"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090916.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090916.b"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090915.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090915.b"
    }
  },
  {
    "at": "2026-06-08T16:22:18Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090914.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:17Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090914.b"
    }
  },
  {
    "at": "2026-06-08T16:22:17Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090913.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:22:17Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090913.b"
    }
  },
  {
    "at": "2026-06-08T16:22:17Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "files_detected",
    "details": {
      "count": 10
    }
  },
  {
    "at": "2026-06-08T16:22:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "dependencies_ok",
    "details": {
      "opmet_primary_mode": "RB",
      "opmet_primary_url": "http://opmet_parser:7000",
      "opmet_shadow_mode": "CS",
      "opmet_shadow_url": "http://old_meteo_text_parser:5001"
    }
  },
  {
    "at": "2026-06-08T16:22:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:21:42Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "sources": {
        "montreal_eccc": {
          "fetched": 4,
          "saved": 4,
          "errors": []
        }
      },
      "fetched": 4,
      "saved": 4,
      "errors": []
    }
  },
  {
    "at": "2026-06-08T16:21:40Z",
    "source": "orchestrator.jtwc.tropical_cyclones",
    "status": "partial",
    "action": "ingest_cycle_finished",
    "details": {
      "advisories_count": 0,
      "records_count": 0,
      "saved_count": 0,
      "fetch_errors": [
        {
          "error_class": "Timeout::Error",
          "error_message": "execution expired",
          "product": "abpwweb.txt"
        },
        {
          "error_class": "SocketError",
          "error_message": "DNS resolution is temporarily unavailable for www.metoc.navy.mil",
          "product": "abioweb.txt"
        }
      ]
    }
  },
  {
    "at": "2026-06-08T16:21:38Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": {
      "source_names": [
        "MONTREAL"
      ]
    }
  },
  {
    "at": "2026-06-08T16:21:38Z",
    "source": "orchestrator.jtwc.tropical_cyclones",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:21:25Z",
    "source": "orchestrator.noaa.metar",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 268,
      "skipped": 4942
    }
  },
  {
    "at": "2026-06-08T16:21:23Z",
    "source": "orchestrator.roshydro",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "files_count": 1
    }
  },
  {
    "at": "2026-06-08T16:21:23Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090912.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:21:23Z",
    "source": "orchestrator.noaa.taf",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 53,
      "skipped": 3018
    }
  },
  {
    "at": "2026-06-08T16:21:10Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090912.b"
    }
  },
  {
    "at": "2026-06-08T16:21:10Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "files_detected",
    "details": {
      "count": 1
    }
  },
  {
    "at": "2026-06-08T16:21:08Z",
    "source": "orchestrator.noaa.metar",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:20:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "dependencies_ok",
    "details": {
      "opmet_primary_mode": "RB",
      "opmet_primary_url": "http://opmet_parser:7000",
      "opmet_shadow_mode": "CS",
      "opmet_shadow_url": "http://old_meteo_text_parser:5001"
    }
  },
  {
    "at": "2026-06-08T16:20:08Z",
    "source": "orchestrator.noaa.taf",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:20:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:19:50Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "sources": {
        "tokyo_jma": {
          "fetched": 50,
          "saved": 50,
          "errors": []
        }
      },
      "fetched": 50,
      "saved": 50,
      "errors": []
    }
  },
  {
    "at": "2026-06-08T16:19:22Z",
    "source": "orchestrator.noaa.metar",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 361,
      "skipped": 4841
    }
  },
  {
    "at": "2026-06-08T16:19:21Z",
    "source": "orchestrator.roshydro",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "files_count": 2
    }
  },
  {
    "at": "2026-06-08T16:19:21Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090911.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:19:19Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090911.b"
    }
  },
  {
    "at": "2026-06-08T16:19:19Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090910.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:19:19Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "OPKC",
      "observed_at": "2026-06-08T16:30:00Z",
      "now_utc": "2026-06-08T16:19:19Z"
    }
  },
  {
    "at": "2026-06-08T16:19:16Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090910.b"
    }
  },
  {
    "at": "2026-06-08T16:19:16Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "files_detected",
    "details": {
      "count": 2
    }
  },
  {
    "at": "2026-06-08T16:19:08Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": {
      "source_names": [
        "TOKYO"
      ]
    }
  },
  {
    "at": "2026-06-08T16:19:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "dependencies_ok",
    "details": {
      "opmet_primary_mode": "RB",
      "opmet_primary_url": "http://opmet_parser:7000",
      "opmet_shadow_mode": "CS",
      "opmet_shadow_url": "http://old_meteo_text_parser:5001"
    }
  },
  {
    "at": "2026-06-08T16:19:08Z",
    "source": "orchestrator.noaa.metar",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:19:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:18:35Z",
    "source": "orchestrator.volcanic_ash.advisories",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "sources": {
        "washington_ospo": {
          "fetched": 50,
          "saved": 50,
          "errors": []
        }
      },
      "fetched": 50,
      "saved": 50,
      "errors": []
    }
  },
  {
    "at": "2026-06-08T16:18:24Z",
    "source": "orchestrator.noaa.metar",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 117,
      "skipped": 5102
    }
  },
  {
    "at": "2026-06-08T16:18:24Z",
    "source": "orchestrator.noaa.taf",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "messages_count": 17,
      "skipped": 3027
    }
  },
  {
    "at": "2026-06-08T16:18:05Z",
    "source": "orchestrator.noaa.tropical_cyclones",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "products_count": 16,
      "downloaded_count": 0,
      "parsed_xml_count": 0,
      "records_count": 0,
      "saved_count": 0,
      "raw_payloads_saved": 3,
      "skipped_count": 16,
      "by_basin": {
        "east_pacific": 16
      },
      "by_format": {
        "unknown": 2,
        "shp": 6,
        "kmz": 8
      }
    }
  },
  {
    "at": "2026-06-08T16:18:01Z",
    "source": "orchestrator.roshydro",
    "status": "ok",
    "action": "ingest_cycle_finished",
    "details": {
      "files_count": 1
    }
  },
  {
    "at": "2026-06-08T16:18:00Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "opmet_shadow_processed",
    "details": {
      "filename": "UUUJ03090909.b",
      "opmet_primary_mode": "RB",
      "opmet_shadow_mode": "CS"
    }
  },
  {
    "at": "2026-06-08T16:17:53Z",
    "source": "orchestrator.noaa.tropical_cyclones",
    "status": "info",
    "action": "ingest_cycle_started",
    "details": null
  },
  {
    "at": "2026-06-08T16:17:20Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "LATI",
      "observed_at": "2026-06-08T16:20:00Z",
      "now_utc": "2026-06-08T16:17:20Z"
    }
  },
  {
    "at": "2026-06-08T16:17:20Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "ESNX",
      "observed_at": "2026-06-08T16:20:00Z",
      "now_utc": "2026-06-08T16:17:20Z"
    }
  },
  {
    "at": "2026-06-08T16:17:20Z",
    "source": "metar_ingest",
    "status": "skipped",
    "action": "skip_future_metar",
    "details": {
      "reason": "future_observed_at",
      "source": "roshydro",
      "station": "LWOH",
      "observed_at": "2026-06-08T16:20:00Z",
      "now_utc": "2026-06-08T16:17:20Z"
    }
  },
  {
    "at": "2026-06-08T16:17:17Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "processing_file",
    "details": {
      "filename": "UUUJ03090909.b"
    }
  },
  {
    "at": "2026-06-08T16:17:17Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "files_detected",
    "details": {
      "count": 1
    }
  },
  {
    "at": "2026-06-08T16:17:08Z",
    "source": "orchestrator.roshydro",
    "status": "info",
    "action": "dependencies_ok",
    "details": {
      "opmet_primary_mode": "RB",
      "opmet_primary_url": "http://opmet_parser:7000",
      "opmet_shadow_mode": "CS",
      "opmet_shadow_url": "http://old_meteo_text_parser:5001"
    }
  }
]