CM4 is missing IEEE1588-2008 support through BCM54210PE

[tschiemer]

Is this the right place for my bug report?
Yes: I asked on the forum, I asked the raspberry foundation through the contact form but didn't get any response - it is very much kernel related as is explained in the forum post.

Describe the bug
The CM4 has an undocumented inofficial PHY (BCM54210PE) that supports hardware timestamping (and a hardware clock, actually??), but there seems to be no driver supporting these features.

So CM4 does actually not provide hardware based IEEE1588-2008 support as opposed to what the raspberry pi foundation actually communicates in the CM4 datasheet.

To reproduce
sudo ethtool -T eth0 only shows software timestamping capabilities.

Expected behaviour
Depends - the raspberry pi foundation never communicated what IEEE1588-2008 features are supported such that is is unclear what exactly can be expected; also the PHY documentation is not available such that possibilities (ie optimal expected behaviour) can not be described.

Expected (according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s1-using_ptp):

• eth support for:
  • SOF_TIMESTAMPING_TX_HARDWARE
  • SOF_TIMESTAMPING_RX_HARDWARE
  • (possibly) SOF_TIMESTAMPING_RAW_HARDWARE
• shows PTP Hardware Clock (because BCM54210PE has on-chip clock)
• Hardware Transmit Timestamp Modes ??
• Hardware Receive Filter Modes ??

The CM4 datasheet lists SYNC_IN, SYNC_OUT pins ment for IEEE1588-2008 support but has not further specified this.
It actually is a question wether these pins are needed in the first place - because the PHY would seem to have a hardware-clock itself.
Further it must be clear wether the PHY's internal clock (if any) is routed to a CM4 pin in any form and if not this should be fixed in future revisions because custom boards for the CM4 might require tight synchronization to a hardware clock.

Actual behaviour
sudo ethtool -T eth0 only shows software timestamping capabilities.
There is no (documented) synchronizable IEEE1588-2008 clock signal going from the CM4.

System
Raspberry Pi Compute Module 4

Raspberry Pi OS Lite, January 11th 2021, Kernel version: 5.4
Ubuntu Server 20.04.2 LTS,

Logs
none

Additional context
BCM54210PE documentation is required to implement to see actual features provided and configuration options - IEEE802.3-2018 seems to define said capabilities etc.
Possibly also required are CM4 schematics and BCM2711 documentation.
Also see IEEE802.3-2018 Section 90. Ethernet support for time synchronization protocols

I can try to take care of the implementation, but I would like to have the mentioned documents.

[pelwell]

@ghollingworth?

[tschiemer]

Just compiling a kernel with PTP timestamping support enabled (by default not). Will see if this changes anything.

I'm new to kernel topics, so I missed the obvious. Will report back

[tschiemer]

Built a new kernel according to docs and changed the following settings (through make menuconfig):

CONFIG_NETWORK_PHY_TIMESTAMPING=y

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=m
# CONFIG_DP83640_PHY is not set
# CONFIG_PTP_1588_CLOCK_INES is not set
# CONFIG_PTP_1588_CLOCK_IDT82P33 is not set
# CONFIG_PTP_1588_CLOCK_IDTCM is not set
# end of PTP clock support

Linux cm4-1 5.10.14-v7l-MY_CUSTOM_KERNEL+ #1 SMP Tue Feb 16 14:05:19 GMT 2021 armv7l GNU/Linux

No change observed, ie

pi@cm4-1:~/linux $ ethtool -T eth0
Time stamping parameters for eth0:
Capabilities:
	software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
	software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
	software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none

pi@cm4-1:~ $ sudo lshw -class network
  *-network:0
       description: Ethernet interface
       physical id: 2
       logical name: eth0
       serial: dc:a6:32:XX:XX:XX
       size: 1Gbit/s
       capacity: 1Gbit/s
       capabilities: ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=bcmgenet driverversion=5.10.14-v7l-MY_CUSTOM_KERNEL+ duplex=full ip=192.168.2.197 link=yes multicast=yes port=MII speed=1Gbit/s
...

I figure because there is no particular PTP hardware clock (driver) there's no recognized PTP clock (...) - but this would depend on wether BCM54210PE actually would have a hardware clock (?)

bcmgenet does not seem to use phy_devices struct mii_timestamper *mii_ts;

so enabling timestamping has no effect...

For reference: https://github.com/raspberrypi/linux/blob/c9226080e513181ffb3909a905e9c23b8a6e8f62/include/linux/mii_timestamper.h

[ghollingworth]

We're currently discussing getting support for writing a driver from Broadcom, although we have access to the datasheet, to me it's pretty meaningless!

Unfortunately, so is the Linux interface to the clock...

But we'll endeavour to get there

[tschiemer]

Ok, thanks!

Hoping to be hearing more from you!

Just to say, as for the docs, I wouldn't mind signing an NDA to have a look to see if can make sense of it..

[wowczarek]

Damn you Broadcom and your closed datasheets! Timestamping at the PHY level is the best one can get, but not if unimplemented in software...

I also do hope those SYNC_IN and SYNC_OUT pins are actual event inputs/outputs for external sync and were preserved on the I/O board and not repurposed for something else. I've seen countless SBCs where this potential was wasted.

So, @ghollingworth, aside from a PTP driver eventually being developed - do we know if the two pins mentioned (pins 16 and 18 on the CM4) are presented on the I/O board?

Thanks!

[tschiemer]

@wowczarek

From what I see in the CM4IO datasheet (page 7) SYNC_IN, SYNC_OUT are available on J2 pins 8+9

[pelwell]

Damn those open datasheets!

[wowczarek]

@pelwell haha, senior moment :)

[jaycordaro]

I saw @wowczarek message on TimeNuts. I used to work for Broadcom's Ethernet PHY group and worked on the 5421x series including the IEEE1588 part of the chip. I remain on good relations with my former colleagues. There are several packages and two dies which map to the BCM54210 part number. I think the part on the Raspberry Pi (BCM54213PE? is that correct) does not pin out the SYNC_IN and SYNC_OUT pins even though they are on the die (36 pin package). I will check with them. Even if the HW SYNC pins are not brought out, some functionality of the IEEE-1588 portion of the PHY may be able to be enabled.

@ghollingworth, does the datasheet you have access to mention IEEE-1588?

[dp111]

On the CM4 we use the BCM54210PE and the SYNC_IN and SYNC_OUT are exposed to pins on the CM4 connector. The CM4IO board exposes these signals on connector J2. @jaycordaro if the PHY team are looking at a driver for this it would be great.

[jaycordaro]

ok I spoke to some friends who are still at Broadcom. The PHY does support HW timestamping, the register writes required to enable HW timestamping and the SYNC pins are described in an applications note, "IEEE-1588 for Timing Applications," not in the datasheet, so that may explain why the datasheet didn't provide anything useful. Someone who has Broadcom DocSafe access for BCM54210PE could request that document to be added to their account. The Product Line Manager would then approve the request. The PHY team people that I talked to (senior ASIC mgr for the PHY and someone in Marketing) are not aware of any effort inside the company to support a driver but they will try to help.

[ghollingworth]

Thanks to everyone on this... Especially to Philphil on the forums and @jaycordaro for their help! I do have access to the register information and Broadcom Docsafe and am currently trying to play catchup with the PTP kernel interface and ioctl interface to understand what it is I need to implement to get a working timestamping system.

What's clear is that there are a number of interfaces implementing different functionality, just need to decide on the order of implementation.

What is it that people actually want out of this?

1. external signal timestamping (timestamping of the SYNC_IN signal)
2. external event creation (SYNC_OUT periodic output signal)
3. TX and RX packet hardware timestamping (the BRCM MAC contains a FIFO from which you can read hardware timestamps)
4. PTP Hardware clock (which I think is implied with 1 and 2 but maybe not 3

Gordon

[wowczarek]

@ghollingworth actually 4. is implied exactly in 3., at least from the kernel interface point of view, for network sync (IEEE 1588) to work.

What is needed is all of the items you list, but as to the order:

a. 4. is required because this is the interface to controlling the clock.
b. 3. is required to sync this clock over the network
c. 1. is required to sync with an external hardware pulse such as from GNSS receivers or other quality clocks
d. 2. is required for external measurement and characterisation, which is priceless

• Items 1. and 2. are rare opportunities for hardware this common, giving this platform a chance to be the prime off-the-shelf testing ground for time sync related projects (*)
• Items 3. and 4. are the part that any other 1588-compliant chip has driver support for.

So unselfishly I'd say 3,4, then 1,2, but really it's the whole that people / we are / should be after.

(*) While we are at it, a set of u.fl (or ideally SMA) connectors could be considered for these two pins in the next revision of the carrier board instead, and freeing up the pins for something else. That would mean plug and play for serious timing, but it's not a big issue otherwise, we have the header. This is debatable though - a GNSS receiver on a board will also have a pin, but a more serious GNSS driven oscillator will have a coax connector.

Thanks!

[ghollingworth]

Thanks that helps a lot.

If you've got a suggestion for an Ethernet device that does actually implement these functions from the standard set of drivers:

I would suggest one of these (i.e. drivers that reference the ptp_clock_register function)

https://elixir.bootlin.com/linux/latest/C/ident/ptp_clock_register

That would help me with an initial implementation...

Also how would you test it? I can run ptp4l which I'm assuming would test the hardware timestamping, but may not test the hardware clock interface...