LCP terminated by peer

Bruce Ediger

2024-10-20 (Last Modified: 2024-10-20)

I run pacman -Syu about once a week on my Arch Linux machines, to stay up-to-date, avoid security problems and generally stay at the bleeding edge of software revisions.

Today, I ran pacman -Syu on my Dell R530 server and it updated the PPP package.

Trouble ensued.

I happened to notice that the ppp package got updated today, from v2.5.0-3 to v2.5.1-1. My server’s been running ppp v2.5.0-3 since 2024-01-06, and there was a problem from that update, too. I seem to recall meddling with /etc/ppp/peers/centurylink, but I didn’t write down what I did that time. Nevertheless, I was alert for problems after I did the ritual reboot after updating.

It takes about 2 minutes from typing sudo systemctl reboot to logging in to a freshly rebooted R530 server. During those 2 minutes, lots of stuff happens, including setting up a Point to Point Protocol link to my ISP, currently CenturyLink. That’s where the ppp and rp-pppoe packages come in.

After finishing rebooting, no network connection to CenturyLink existed. I looked in the log to see what was them matter, with journalctl -f -u mypppd.service

Oct 20 17:41:22 monarch systemd[1]: Started PPP link to Centurylink.
Oct 20 17:41:22 monarch pppd[882]: Plugin /usr/lib/rp-pppoe/rp-pppoe.so loaded.
Oct 20 17:41:22 monarch pppd[882]: Plugin /usr/lib/rp-pppoe/rp-pppoe.so loaded.
Oct 20 17:41:22 monarch pppd[882]: RP-PPPoE plugin version 4.0 compiled against pppd 2.5.1
Oct 20 17:41:22 monarch pppd[882]: RP-PPPoE plugin version 4.0 compiled against pppd 2.5.1
Oct 20 17:41:22 monarch pppd[882]: pppd 2.5.1 started by root, uid 0
Oct 20 17:41:37 monarch pppd[882]: PPP session is 4164 (0x1044)
Oct 20 17:41:37 monarch pppd[882]: Connected to 78:fe:3d:a3:27:c2 via interface eno3.201
Oct 20 17:41:37 monarch pppd[882]: Using interface ppp0
Oct 20 17:41:37 monarch pppd[882]: Connect: ppp0 <--> eno3.201
Oct 20 17:41:37 monarch pppd[882]: peer from calling number 78:FE:3D:A3:27:C2 authorized
Oct 20 17:41:37 monarch pppd[882]: LCP terminated by peer
Oct 20 17:41:40 monarch pppd[882]: Connection terminated.
Oct 20 17:41:40 monarch pppd[882]: Sent PADT
Oct 20 17:41:40 monarch pppd[882]: Modem hangup
Oct 20 17:42:10 monarch pppd[882]: PPP session is 4286 (0x10be)
Oct 20 17:42:10 monarch pppd[882]: Connected to 78:fe:3d:a3:27:c2 via interface eno3.201
Oct 20 17:42:10 monarch pppd[882]: Using interface ppp0
Oct 20 17:42:10 monarch pppd[882]: Connect: ppp0 <--> eno3.201
Oct 20 17:42:10 monarch pppd[882]: peer from calling number 78:FE:3D:A3:27:C2 authorized
Oct 20 17:42:10 monarch pppd[882]: LCP terminated by peer
Oct 20 17:42:13 monarch pppd[882]: Connection terminated.
Oct 20 17:42:14 monarch pppd[882]: Sent PADT
Oct 20 17:42:14 monarch pppd[882]: Modem hangup
Oct 20 17:42:44 monarch pppd[882]: PPP session is 4286 (0x10be)
...

pppd, the actual PPP process, was looping like that. The relevant error message appeared to be LCP terminated by peer

Googling for that phrase led me nowhere. There’s several references to be found on the compendium of human knowledge, the World Wide Web, but they didn’t seem to apply.

Enabling “MPPE” in some Windowsy point-n-click dialog box I put mppe-stateful in /etc/ppp/peers/centurylink No joy.
Don’t know what this one means.
???

I did glean that putting debug in /etc/ppp/peers/centurylink might show more in the systemd journal, and it does. Unfortunately, what extra logging it shows isn’t helpful

I checked in /etc/ppp/, the configuration directory for pppd. I noticed that the PPP (or PPPoE) login credentials files were only kind of present:

-rw------- 1 root root    78 Sep 19 14:24 chap-secrets.example
-rw------- 1 root root   117 Jan  4  2021 chap-secrets.pacsave
-rw------- 1 root root    77 Sep 19 14:24 pap-secrets.example
-rw------- 1 root root   116 Jan  4  2021 pap-secrets.pacsave

The Arch Linux package manager, pacman, will save copies of files if it installs new versions. Those are the chap-secrets.pacsave and pap-secrets.pacsave files. I didn’t recognize this immediately, and did some paniced searching for copies I might have saved. I finally remembered that .pacsave is the suffix pacman puts on config files it might overwrite when installing a new package version.

The solution:

sudo cp /etc/ppp/pap-secrets.pacsave /etc/ppp/pap-secrets
sudo cp /etc/ppp/chap-secrets.pacsave /etc/ppp/chap-secrets
sudo systemctl restart mypppd.service

The mppe-stateful entry in /etc/ppp/peers/centurylink did not matter. I took it out, restarted pppd, and everything still works.

The failure of pppd to set up a connection to CenturyLink also deeply confused dnsmasq, which I’m using for DNS aggregation.

journalctl -u dnsmasq shows a whole mess of problems like:

Oct 20 17:42:46 monarch dnsmasq[927]: query[A] ssl.gstatic.com.glamour from 10.0.20.35
Oct 20 17:42:46 monarch dnsmasq[927]: config error is REFUSED (EDE: network error)

dnsmasq also tried to send A and AAAA queries to the configured DNS servers that it should have answered out of its own cache and local DNS config.

Oct 20 19:49:40 monarch dnsmasq[927]: query[A] monarch.glamour from 10.0.20.35
Oct 20 19:49:40 monarch dnsmasq[927]: forwarded monarch.glamour to 9.9.9.9
Oct 20 19:49:40 monarch dnsmasq[927]: forwarded monarch.glamour to 8.8.8.8
Oct 20 19:49:40 monarch dnsmasq[927]: query[AAAA] monarch.glamour from 10.0.20.35
Oct 20 19:49:40 monarch dnsmasq[927]: forwarded monarch.glamour to 8.8.8.8
Oct 20 19:49:40 monarch dnsmasq[927]: reply monarch.glamour is NXDOMAIN
Oct 20 19:49:40 monarch dnsmasq[927]: reply monarch.glamour is NXDOMAIN

monarch.glamour is 10.0.0.1 on my LAN. dnsmasq knows this because that name → IP address matchup is in /etc/hosts on the Dell R530. I think this is due to no PPP connection, no route, to any big DNS server like 8.8.8.8 or 9.9.9.9.

The PPP-over-ethernet plugin, rp-pppoe, got updated 2024-09-20 without me even noticing. Between ppp and rp-pppoe, this connection to CenturyLink seems to be on shakey metaphysical ground.