Question: can we collect statistic if the table doesn't collect statistic during creating

[xudong963]

I noticed that after creating a table, enabling collect_stat, and scanning the table, we won't get the statistics for the table.

We must enable collect_stat before creating a table, then when scan the table, we can get the statistics.

set datafusion.execution.collect_statistics = true;

CREATE EXTERNAL TABLE t2 (id INT not null, date DATE) STORED AS PARQUET LOCATION './data/' PARTITIONED BY (date) WITH ORDER (id ASC);

INSERT INTO t2 VALUES (4, '2025-03-01'), (3, '2025-3-02'), (2, '2025-03-03'), (1, '2025-03-04');

SELECT * FROM t2 ORDER BY id ASC; // we'll get the statistics

CREATE EXTERNAL TABLE t2 (id INT not null, date DATE) STORED AS PARQUET LOCATION './data/' PARTITIONED BY (date) WITH ORDER (id ASC);

INSERT INTO t2 VALUES (4, '2025-03-01'), (3, '2025-3-02'), (2, '2025-03-03'), (1, '2025-03-04');

set datafusion.execution.collect_statistics = true;

SELECT * FROM t2 ORDER BY id ASC; // we won't get the statistics

Do we have ways to get statistics if we don't enable collect_statistics before creating the table?

[xudong963]

Some dbs can using analyze table command to update or collect statistics after creating the table

[alamb]

Do we have ways to get statistics if we don't enable collect_statistics before creating the table?

TLDR is I don't know of any way to get statistics after creating the table with the built in SQL API. Analyze table is an interesting approach. The alternate is to implement such semantics outside of the DataFusion core

[alamb]

Actually, I wonder if it would be reasonable to update ListingTable so that it would collect statistics if they were not already present, when datafusion.execution.collect_statistics is true.

I have personally found the current situation where the setting only applies on creation to be pretty confusing, honestly