[RPKI] routinator-0.13.1 core dump

[Robert Blayzor]

I have been running into stability issues since the release of 0.13 and 
it's been getting worse. I'm having routinator servers crash multiple 
times a day now and I just had two servers crash basically at the same 
time, so it's getting some outside information that it doesn't like.

The servers are running FreeBSD 14.0 and Rust 1.76 is installed on the 
system. This has been happening since 0.13 and is now still happening in 
0.13.2, but now even more frequently it appears.

Unfortunately when routinator core dumps, it's not leaving a core file 
behind, I'm just seeing:

pid 2982 (routinator), jid 0, uid 245: exited on signal 6 (no core dump 
- bad address)
pid 93548 (routinator), jid 0, uid 245: exited on signal 6 (no core dump 
- bad address)
pid 75744 (routinator), jid 0, uid 245: exited on signal 6 (no core dump 
- bad address)


Literally the last noise in the logs on each server. This level of 
"noise" and failure looks like something is broken....


Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/EA021982D2C311EE98D88865775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/6667703ED47D11EEAA6502BE775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/86D788C8D48111EE80528CC4775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/EFBE9CB6D16911EEA51004A8775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/8668158AD17311EEB69D94B8775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/B7C19E14D2CF11EEBB636E78775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/B8A5AEA0D48311EE85B40BC8775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/0B3122E8D17911EE951057C1775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/385F1D8AD2D011EE9BBD5C79775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 routinator[75742]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/ED2CF494D2C611EEAA8E776A775412E6.roa: certificate has 
expired.
Mar 16 12:43:43 rs0 kernel: pid 75744 (routinator), jid 0, uid 245: 
exited on signal 6 (no core dump - bad address)




Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/11A8344ED2C411EEB833F865775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/3E62BCE0D2C711EEAEF6EA6A775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/C5EC9C26D2CC11EE8E67C973775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/8C32E506D2CE11EEA31CD576775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/F8491F0ED2C811EE8D92D66D775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/1543656CD2C711EE9822A16A775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/2040974A0F1911EE83AEF94F4AD9E6FC.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/A940268AD15D11EE8290C094775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 routinator[95204]: 
rsync://rpki.afrinic.net/repository/member_repository/F368F2D0/92F86E1C6E0511E8A1
B5854BF8AEA228/5812BEAED20111EE9D9842A6775412E6.roa: certificate has 
expired.
Mar 16 08:48:48 rs0 kernel: pid 95206 (routinator), jid 0, uid 245: 
exited on signal 6 (no core dump - bad address)