IDLE: turn built-in extensions into regular modules

[terryjreedy]

BPO	27099
Nosy	@rhettinger, @terryjreedy, @serhiy-storchaka, @mlouielu, @csabella, @wohlganger
PRs	bpo-27099: IDLE - Convert built-in extensions to regular features #2494 [3.6] bpo-27099: IDLE - Convert built-in extensions to regular featur… #3487 bpo-27099: Finish updating IDLE doc and help text. #3510 [3.6] bpo-27099: Finish updating IDLE doc and help text. (GH-3510) #3511
Dependencies	bpo-24225: Idlelib: changing file names bpo-27173: Modern Unix key bindings for IDLE bpo-30779: IDLE: configdialog -- factor out Changes class bpo-30781: IDLE: configdialog -- switch to ttk widgets.
Files	config-extensions.def config-main.def 2017-07-07 13_58_45-Settings.png: setting screenshot - highlight tab.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/terryjreedy'
closed_at = <Date 2018-01-23.04:14:42.619>
created_at = <Date 2016-05-24.02:02:40.067>
labels = ['expert-IDLE', 'type-bug', '3.7']
title = 'IDLE: turn built-in extensions into regular modules'
updated_at = <Date 2018-01-23.07:10:19.842>
user = 'https://github.com/terryjreedy'

bugs.python.org fields:

activity = <Date 2018-01-23.07:10:19.842>
actor = 'terry.reedy'
assignee = 'terry.reedy'
closed = True
closed_date = <Date 2018-01-23.04:14:42.619>
closer = 'rhettinger'
components = ['IDLE']
creation = <Date 2016-05-24.02:02:40.067>
creator = 'terry.reedy'
dependencies = ['24225', '27173', '30779', '30781']
files = ['43384', '43385', '46996']
hgrepos = []
issue_num = 27099
keywords = ['patch']
message_count = 49.0
messages = ['266214', '266221', '266778', '267575', '268479', '268481', '268511', '268517', '297450', '297451', '297904', '297918', '298160', '298176', '298183', '298225', '298253', '298687', '298690', '298723', '298740', '298982', '299011', '299015', '299025', '299074', '299153', '300793', '300798', '300804', '300807', '300809', '300813', '300822', '301008', '301159', '301160', '301178', '301182', '301833', '301842', '301847', '301850', '301951', '301954', '302715', '310466', '310467', '310475']
nosy_count = 6.0
nosy_names = ['rhettinger', 'terry.reedy', 'serhiy.storchaka', 'louielu', 'cheryl.sabella', 'wohlganger']
pr_nums = ['2494', '3487', '3510', '3511']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27099'
versions = ['Python 3.6', 'Python 3.7']

[terryjreedy]

Followup to bpo-24225, which renamed idlelib files.

Currently, an arbitrary set of 9 (of 10, if Check Module is counted as a separate feature) IDLE features are implemented as extensions. Their menu names are as follows:
Edit - Show Completions
Edit - Expand Word
Edit - Show call tip
Edit - Show surrounding parens
Format - Format Paragraph
Format - Strip trailing whitespace
Run - Check Module (same file as below)
Run - Run Module
Options - Code Context
Window - Zoom Height

This issue proposes to turn these into normal features implemented in regular code.

1. The menu entries for extensions are not included in the main menu definition file (mainmenu, previously Bindings). Rather, they are appended when the file is read. This is constricting. The order of the 4 edit entries is the alphabetical order of the corresponding file names. Run Module should be at the top of the Run menu, rather than being shoved down further if and when more non-extension entries are added.

2. Each feature is a separate file, usually small to medium with one class. This bloats the number of modules in idlelib (now about 60). I am considering combining autoexpand and parenmatch, paragraph and rstrip, and codecontext and zoomheight (and moving codecontext to the Window menu). They might be joined with other features from the same menu.

3. The user customization system is not very compatible with change. The problems with extension customization and changing file names is discussed in the second half of msg266213. If extensions are not converted, another solution will be needed.

I have not yet tested the effect of adding new key bindings to config-keys.def and config-keys.cfg. If this causes a problem for current releases, then this is a problem we need to face. I anticipate wanting to add other new keybindings, such as one to switch tabs when we use a notebook.

[ned-deily]

I don't have an opinion on the matter, other than a reminder about the current cross-version (and cross-platform) compatibility issues with a user's IDLE configuration files as alluded to in bpo-20580. Perhaps others more familiar with IDLE's internals would have comments?

[terryjreedy]

The version conflict problem is why I reverted changing 'extension names' to match the new file names, before rebase collapsing my patches. I was really glad most were in a separate patch.

Each time a feature ceases to be an extension, the (old) extension name will have to be added to a set of names ignored when calculating the set of extensions. I will start with one until I have worked out and tested the details.

[terryjreedy]

The final paragraph of my initial post should have talked about config-main and features, rather than keys.

The following experiment indicates that adding new sections to config-main.def and customizations thereof to config-main.cfg should not be a problem.

Add the following to config-main.cfg
[NewSection]
new = True

Open IDLE 2.7 from console (python, with import).
Open config dialog.
Change item.
Close with [OK]
confirm that change is written and new section is left alone.
Close IDLE.
Repeat with 3.5.

Old versions will not see new config-main.def, so that is not an issue.

Conclusion: adding a new section that old versions ignore is much safer then adding a new value for a current section&item that old versions do read and act on, and which may refer to something that does not exist. (This was the problem with IDLE Dark theme and would be with Unix New keyset).

The remaining issues:

4. Finding a place on the dialog itself for new customization widgets. On Font, Theme, and General tabs, there is space available either now or with some changes. The dialog can be enlarged if needed.

5. Adding the behind-the-scenes plumbing. This is mostly straightforward.

6. Doing minimal refactoring to make better testing easier.
   That would include being able to patch in a .idlerc directory or individual user files. The files are small and could easily fit in memory as StringIOs.

I would like to be able to set user files, open config and idleConf, open config dialog, simulate user actions on the dialog, close, and check the effect on the files.

[terryjreedy]

The first line of the last message should have said 'in addition to' instead of 'rather than', In the following expanded table, Key is the number of pseudoevents with configurable and fixed key bindings. Gen is the number of General pseudoevents. (Current, from config-main.def.)

File Menu Key Gen
------------ ------------------------------ --- ---
autocomplete Edit - Show Completions 1,2 1
autoexpand Edit - Expand Word 1,- -
calltips Edit - Show call tip 1,2 -
parenmatch Edit - Show surrounding parens 1,1 3
paragraph Format - Format Paragraph 1,- 1
rstrip Format - Strip trailing white- -,- -
runscript Run - Check Module 2,0 -
runscript Run - Run Module same file
codecontext Options - Code Context -,- 2
zoomheight Window - Zoom Height 1,- -

My initial experiment with adding a fake event and binding in the user config-keys.cfg does not affect existing versions. It appears to be completely ignored and does not appear on the list of configurable keys in the dialog.

[terryjreedy]

Ned, I reread bpo-20580. It reminds me that both Control and Alt are problems. In built-in config-extensions.def, the builtin fixed bindings and one of the configurable bindings are:

autocomplete: <Key-Tab> <KeyRelease-period> <KeyRelease-slash> <KeyRelease-backslash>
calltips: <KeyRelease-parenleft> <KeyRelease-parenright> <KeyRelease-0>
parenmatch: <KeyRelease-parenright> <KeyRelease-bracketright> <KeyRelease-braceright>
runscript: <Key-F5> (configurable)

I presume that these all work fine on Macs. Every other configurable binding uses 'Control' or 'Alt'. When I copy configurable bindings into the file, I have to put them in each section. Should I change each to 'Command' or 'Option' for the 'Mac' and 'Osx' sections? Also see comment on bpo-20580.

I am adding bpo-27173 as a dependency, for now, because it might be easier to include the new key set before starting to add new bindings.

[terryjreedy]

Ned, from your response on 20580, there appears to be no conversion rule. Perhap you could go throuch config-main.def and make a list of what you want for Mac and Unix respectively, if different from the current binding. Serhiy, should anything be different for Linux?, or for your Modern Linux?

[ned-deily]

Terry, I don't know that much about the history of the config files. I do know that during framework installs of Python on OS X, the "install_IDLE" recipe in Mac/Makefile.in (which ./configure uses to produce a configured Mac/Makefile) has some editing steps, using sed, to alter the IDLE .def files as they are being installed. I'm attaching the two edited .def files as installed by the OS X installer. And then there's the runtime munging of "Alt-" to "Option-" in GetCurrentKeySet() of idlelib/config.py.

[terryjreedy]

I am thinking that we should maybe add an 'Editor' tab. Or we could expand the dialog box. More room on General tab can be obtained by removing the label frames, or at least the labels, which seem a bit redundant.

[terryjreedy]

This has to wait for at least some of the refactoring and increased testing of configdialog, and maybe for the switch to ttk.

[wohlganger]

Progress Update: I've moved all of the basic functionality of the extensions into the regular parts of IDLE, including menus and keyboard shortcuts. parenmatch and codecontext have all of their settings now in the Highlighting settings tab. I have added theme elements for both, as that was both easier and more in-line with what the rest of the IDLE settings were doing than trying to have a separate color-picker for them. The attached image is what it looks like.

What is left for me to do: add the configuration dialogs and proper setting saving for the remaining extensions/modules. Change help texts. Once the configdialog changes patch (bpo-30779) goes through, I will of course, need to fix anything both touch, but that shouldn't be too difficult.