Flatcar on EC2 sets a incorrect os hostname

[jkroepke]

Description

By default, on flatcar (Alpha?), the hostname contains dots and systemd-networkd is not set hostname received through DHCP.

Readings the docs at https://www.freedesktop.org/software/systemd/man/hostnamectl.html, a system hostname should not contain any dots and should be a single dns label.

Also Debian follows this pattern.

On Flatcar, the /etc/hostname is populated by coreos-metadata. For AWS, the change is introduced in flatcar/bootengine#21, but also mention that systemd-hostname will set the hostname, which does not happen.

I also dump a DHCP traffic which confirms that the hostname is delivered by DHCP.

tcpdump DHCP traffic

# tcpdump -i eth0 -pvn port 67 and port 68
dropped privs to tcpdump
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:25:00.866515 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 0a:5c:80:ef:c6:c0, length 300, xid 0x9ea9fc3e, Flags [none]
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: Discover
      Parameter-Request Option 55, length 13:
        Subnet-Mask, BR, Time-Zone, Classless-Static-Route
        Domain-Name, Domain-Name-Server, Hostname, YD
        YS, NTP, MTU, Option 119
        Default-Gateway
09:25:00.866565 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    10.110.64.1.67 > 10.110.71.54.68: BOOTP/DHCP, Reply, length 548, xid 0x9ea9fc3e, Flags [none]
    Your-IP 10.110.71.54
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: Offer
      Server-ID Option 54, length 4: 10.110.64.1
      Lease-Time Option 51, length 4: 3600
      Subnet-Mask Option 1, length 4: 255.255.224.0
      BR Option 28, length 4: 10.110.95.255
      Domain-Name Option 15, length 29: "eu-central-1.compute.internal"
      Domain-Name-Server Option 6, length 4: 10.110.0.2
      Hostname Option 12, length 19: "i-0b4cb6bd27739f893"
      MTU Option 26, length 2: 9001
      Default-Gateway Option 3, length 4: 10.110.64.1
09:25:00.866818 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 0a:5c:80:ef:c6:c0, length 300, xid 0x9ea9fc3e, Flags [none]
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: Request
      Server-ID Option 54, length 4: 10.110.64.1
      Requested-IP Option 50, length 4: 10.110.71.54
      Parameter-Request Option 55, length 13:
        Subnet-Mask, BR, Time-Zone, Classless-Static-Route
        Domain-Name, Domain-Name-Server, Hostname, YD
        YS, NTP, MTU, Option 119
        Default-Gateway
09:25:00.866855 IP (tos 0x0, ttl 255, id 0, offset 0, flags [DF], proto UDP (17), length 576)
    10.110.64.1.67 > 10.110.71.54.68: BOOTP/DHCP, Reply, length 548, xid 0x9ea9fc3e, Flags [none]
    Your-IP 10.110.71.54
    Client-Ethernet-Address 0a:5c:80:ef:c6:c0
    Vendor-rfc1048 Extensions
      Magic Cookie 0x63825363
      DHCP-Message Option 53, length 1: ACK
      Server-ID Option 54, length 4: 10.110.64.1
      Lease-Time Option 51, length 4: 3600
      Subnet-Mask Option 1, length 4: 255.255.224.0
      BR Option 28, length 4: 10.110.95.255
      Domain-Name Option 15, length 29: "eu-central-1.compute.internal"
      Domain-Name-Server Option 6, length 4: 10.110.0.2
      Hostname Option 12, length 19: "i-0b4cb6bd27739f893"
      MTU Option 26, length 2: 9001
      Default-Gateway Option 3, length 4: 10.110.64.1

If systemd-networkd would set the hostname delivered by DHCP, than the hostname would be correctly set.

Impact

After a successful boot, the hostname of the machine contains dots, e.g. is a FQDN hostname.

# hostname -f && hostname && cat /etc/hostname
i-00c64910e1b14a8ef.eu-central-1.compute.internal
i-00c64910e1b14a8ef.eu-central-1.compute.internal
i-00c64910e1b14a8ef.eu-central-1.compute.internal

Since kubelet use this hostname as default node name, kubelet will register nodes also as an FQDN.
Starting with RBN hostname, AWS related tools in the kubernetes environment (e.g. AWS Cloud Provider) expect the short hostname as node name. See: kubernetes/cloud-provider-aws#335

As workaround, I have to set --hostname-override on all kubernetes components, but this should not be the case on a Container optimized linux system.

Environment and steps to reproduce

1. Set-up: Setup EC2 instance with Flatcar
2. Task: Boot and login into it.
3. Action(s): [ sequence of actions that triggered the bug, see example below ]
   a. cat /etc/hostname
   b. run hostname
4. Error: Contains FQDN instead a dns single-label hostname

Expected behavior

Taken from an Ubuntu instance:

ubuntu@ip-172-16-0-60:~$ hostname -f && hostname && cat /etc/hostname
ip-172-16-0-60.eu-central-1.compute.internal
ip-172-16-0-60
ip-172-16-0-60

Additional information

I'm running flatcar Alpha

Flatcar Container Linux by Kinvolk alpha 3185.0.0 for Amazon EC2

The upstream coreos afterburn will not set the hostname service, if the platform is AWS. See: https://github.com/coreos/afterburn/blob/main/dracut/30afterburn/afterburn-hostname.service#L4

No clue, if systemd will set a transient hostname, if a static hostname is already defined.

[jkroepke]

I also enable debug log of networkd https://superuser.com/a/1234599

# cat /etc/systemd/system/systemd-networkd.service.d/10-debug.conf
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

I can verify, that transient hostname is still set by systemd-networkd, but since the /etc/hostname is populated through coreos metadata, the transient hostname does not matters.

Apr 06 10:46:37 i-08115ab723fbe0417.eu-central-1.compute.internal systemd-networkd[184971]: Setting transient hostname: 'i-08115ab723fbe0417'

I guess rollback flatcar/bootengine#21 would resolve the issue.

Otherwise, afterburn should correctly set the hostname from IMDS.

[jkroepke]

Hi,

running

hostnamectl set-hostname ""
systemctl restart systemd-networkd

Resolve the issue.

# hostname -f && hostname && cat /etc/hostname
i-0f443d994d87d9648.eu-central-1.compute.internal
i-0f443d994d87d9648
cat: /etc/hostname: No such file or directory

This is the current workaround, that I use for now.

systemd:
  units:
    - name: "fix-hostname.service"
      enabled: true
      # https://superuser.com/a/1005804
      contents: |
        [Unit]
        Description=Fix Hostname
        Before=network-pre.target
        Wants=network-pre.target
        
        DefaultDependencies=no
        Requires=local-fs.target
        After=local-fs.target
        
        [Service]
        Type=oneshot
        ExecStart=/usr/bin/rm /etc/hostname
        RemainAfterExit=yes
        
        [Install]
        WantedBy=network.target

[tormath1]

Hi,

Thanks for raising this issue - I think it should be solved with this commit: coreos/afterburn@860db5c which is about to be merged (see: flatcar-archive/coreos-overlay#1769).

The hostname will be truncated to the first dot, so in your case: i-00c64910e1b14a8ef.eu-central-1.compute.internal will become i-00c64910e1b14a8ef.

EDIT: In the meantime, you could maybe try to disable flatcar-metadata-hostname.service from the Ignition config ? Let us know.

[jkroepke]

EDIT: In the meantime, you could maybe try to disable flatcar-metadata-hostname.service from the Ignition config ? Let us know.

How I can disable services that runs inside initramfs? I would expect, that ignition configure only unit files in /sysroot, not configure services inside initramfs.

On a bootet system, there is such no service.

$ systemctl status flatcar-metadata-hostname.service
Unit flatcar-metadata-hostname.service could not be found.

The ability to disable flatcar-metadata-hostname.service would be great.

[jkroepke]

Hi @tormath1

i looked into coreos/afterburn@860db5c . As I understand, the hostname is truncated only if the FQDN have more than 64 bytes. This is not the case here. Even on us-east-1, since the FQDN does not include the region.

in conclusion, flatcar-archive/coreos-overlay#1769 would not help here, EXCEPT https://github.com/coreos/afterburn/blob/main/dracut/30afterburn/afterburn-hostname.service is used for the flatcar-metadata-hostname service, which is not running on AWS by default.

[pothos]

Sounds like we should either exclude ec2 again or throw away the existing shell logic and adapt the new afterburn unit to set the hostname?