Make integration of custom opcodes easier

[x87]

CLEO and its successors add many new custom opcodes (see #71 for example). Currently in order to compile the script you have to alter the original *SCM.INI and add new definitions manually. This is error-prone and unfriendly to new users. During Sanny updates those changes could be lost so you need to update the INI again.

Need to come up with a flexible solution that make Sanny aware of custom opcodes without (ideally) touching its files. This affects the compiler, the disassembler, and IDE (Opcode Search).

[MatiDragon-YT]

These are some of the solutions that I would give you if I could

1. In the same way that modes.xml works, you could do something similar to load new opcodes.txt and SASCM.INI with other names, such as opcodeOfCLEO.txt or newOpc.txt from the mode folder.
   • Personally, I don't like XML because of its novice-unfriendly syntax.
2. Write two new files e.g. XXX and ZZZ, for custom opcodes if they don't exist, but incase that these files if they exist, only read files XXX and ZZZ existing.
3. Could add a tool with an interface similar to "User Tools" and thus make it a little more pleasant.
   

[OrionSR]

Oh wow! Good idea. And is an appropriate issue for the question I had in mind.

Are Restored Opcodes currently supported by Sanny?

Closer to the point, changes to the SCM.ini definitions would influence all of the "command" files, which I've been grouping to include:

• classes.db
• keywords.txt
• opcode.txt (examples.txt)
• scm.ini (opcodes.ini)
• templates.txt
  (I've been considering renaming some files to be more consistent with the XML definitions, and between all games.)
  See: Mobile Modes for GTA3 and Vice City #6 - A little feedback would be appreciated.

For keywords in particular, but also CustomVariables.ini, it would nice to be able to have secondary files with duplicate entries that worked in the compiler, but consistently disassembled to the primary example. #20

Thinking ahead to enumerations; I have found it difficult to maintain a long list in constants.txt built around sheets for each group. I would much prefer to manage these groups as individual files. #40

I agree with @MatiDragon-YT 's assessment of the awkwardness of managing the XML mode files. I have given serious consideration to creating a Modes Manager as a learn-to-GUI project with my son, but that's not going to happen any time soon. Neither of us has done this before and, well... he agreed but has shown no interest in starting. I think a better strategy is to allow the user, or ToolPacks, to update files in a manner that doesn't need to modify the XML file.

<opcodes>@sb:\data\vc\opcodes.ini</opcodes>
<opcodes>@sb:\data\vc\plugins\*.opcodes.ini</opcodes>
<enums>@sb:\data\vc\enum\*.enum.txt</enums>

Goal: To update the edit mode the user would only need to place a file with the appropriate naming convention into the proper folder.

[MatiDragon-YT]

Well.. it would not be bad to start like this.
in the future, maybe I can generate a simple console program that does that job, that of installing new extensions to XML.

[x87]

A GUI tool that allows to add/remove new entries to modes.xml by just dropping a new file onto the tool window and/or unticking a checkbox to temporarily exclude the file would be really handy.

I agree XML is not the most human-friendly format, yet it's much easier comparing to awkward custom syntax of the classes.db.

[OrionSR]

Before I forget again... I was not able to find an error case for the same keyword text being assigned to different opcode numbers.

I think an Edit Mode Manager tool is an awesome idea worthy of it's own discussion, but doesn't solve the stated goal of:

Need to come up with a flexible solution that make Sanny aware of custom opcodes without (ideally) touching its files.

Suggestion for a Mode Manager progression:

• V1 - A tool for switching the current modes.xml file. User Tools appear to be limited to 9 entries. I'm already using 5 just for this purpose. This version would not need to modify xml files.
• V2 - Ability to copy, paste, disable. delete and sort entire mode definitions. For example. copy a PS2 edit mode from a custom root mode to Sanny's default root mode. Or, duplicate a standard edit mode and update the the title and IDs to support different game paths needed for multiple installations of the same game.
  -V3 - A full featured Mode Editor. I figure that mode editing will always be an advanced skill. Ideally, a novice would never need to use the editing features.

[x87]

While working on adding CLEO library and its plugins definitions in different edit modes I realized that providing all possible opcodes by default could be misleading and unintentionally make a scripter assume they use an instruction that is available with vanilla game but in fact this is something that requires an extra piece of software. While it's obvious in context of {$CLEO} scripts that you might want to use CLEO opcodes , it's not like that in main.scm or when you use opcodes introduced with some exotic plugins. You should be aware of such dependency on third-party software and explicitly signal Sanny Builder you want to use it.

Hence comes something new in edit modes - the extensions.

What is an extension? It's a named set of files that define new instructions and related language constructs (keywords, classes).
Each edit mode could have multiple extensions but it must have at least one which would be used with any script. In other words this is what edit modes currently define using <opcodes>, <keywords> , <examples>, etc. So, say, GTA SA v1.0 mode could now look like:

<mode id="sa" title="GTA SA v1.0" game="sa" type="default">
        <data>@sb:\data\sa\</data>

        <extension name="default">
            <opcodes>@sb:\data\sa\SASCM.ini</opcodes>
            <classes>@sb:\data\sa\classes.db</classes>
            <enums>@sb:\data\sa\enums.txt</enums>
            <keywords>@sb:\data\sa\keywords.txt</keywords>
            <constants>@sb:\data\sa\constants.txt</constants>
            <templates>@sb:\data\sa\templates.txt</templates>
            <examples>@sb:\data\sa\opcodes.txt</examples>
        </extension>
        
        <variables>@sb:\data\sa\CustomVariables.ini</variables>
        <labels>@sb:\data\sa\CustomLabels.ini</labels>
        <arrays>@sb:\data\sa\CustomArrays.ini</arrays>
        <missions>@sb:\data\sa\missions.txt</missions>
        <text format="sa">@game:\text\american.gxt</text>
        <ide base="@game:\">@game:\data\default.dat</ide>
        <ide>@game:\data\maps\veh_mods\veh_mods.ide</ide>
    </mode>

In addition to the default extension the mode could define custom extensions, e.g. one for standard CLEO opcodes:

<mode id="sa" title="GTA SA v1.0" game="sa" type="default">
        <data>@sb:\data\sa\</data>

        <extension name="default">
            <opcodes>@sb:\data\sa\SASCM.ini</opcodes>
            <classes>@sb:\data\sa\classes.db</classes>
            <enums>@sb:\data\sa\enums.txt</enums>
            <keywords>@sb:\data\sa\keywords.txt</keywords>
            <constants>@sb:\data\sa\constants.txt</constants>
            <templates>@sb:\data\sa\templates.txt</templates>
            <examples>@sb:\data\sa\opcodes.txt</examples>
        </extension>

        <extension name="cleo">
            <opcodes>@sb:\data\sa\cleo4.ini</opcodes>
            <examples>@sb:\data\sa\cleo4.opcodes.txt</examples>
        </extension>

        <variables>@sb:\data\sa\CustomVariables.ini</variables>
        <labels>@sb:\data\sa\CustomLabels.ini</labels>
        <arrays>@sb:\data\sa\CustomArrays.ini</arrays>
        <missions>@sb:\data\sa\missions.txt</missions>
        <text format="sa">@game:\text\american.gxt</text>
        <ide base="@game:\">@game:\data\default.dat</ide>
        <ide>@game:\data\maps\veh_mods\veh_mods.ide</ide>
    </mode>

or any other CLEO plugin:

<mode id="sa" title="GTA SA v1.0" game="sa" type="default">
        <data>@sb:\data\sa\</data>

        <extension name="default">
            <opcodes>@sb:\data\sa\SASCM.ini</opcodes>
            <classes>@sb:\data\sa\classes.db</classes>
            <enums>@sb:\data\sa\enums.txt</enums>
            <keywords>@sb:\data\sa\keywords.txt</keywords>
            <constants>@sb:\data\sa\constants.txt</constants>
            <templates>@sb:\data\sa\templates.txt</templates>
            <examples>@sb:\data\sa\opcodes.txt</examples>
        </extension>

        <extension name="cleo">
            <opcodes>@sb:\data\sa\cleo4.ini</opcodes>
            <examples>@sb:\data\sa\cleo4.opcodes.txt</examples>
        </extension>

        <extension name="cleo+">
            <opcodes>@sb:\data\sa\cleo+.ini</opcodes>
            <examples>@sb:\data\sa\cleo+.opcodes.txt</examples>
        </extension>

        <variables>@sb:\data\sa\CustomVariables.ini</variables>
        <labels>@sb:\data\sa\CustomLabels.ini</labels>
        <arrays>@sb:\data\sa\CustomArrays.ini</arrays>
        <missions>@sb:\data\sa\missions.txt</missions>
        <text format="sa">@game:\text\american.gxt</text>
        <ide base="@game:\">@game:\data\default.dat</ide>
        <ide>@game:\data\maps\veh_mods\veh_mods.ide</ide>
    </mode>

Now the interesting part. By default the IDE and compiler will only use the default extension. This is something that belong to the game opcodes and is always available for the given game. But if you want to use something extra you should add a new line on top of your script: {$use extension_name}. E.g.:

{$use cleo+}
{$use newOpcodes}

Now both the IDE and compiler know your intention and will provide necessary support (autocomplete, highlighting, compiling). The disassembler will be smart enough to restore list of extensions based on which opcodes are actually used in the file.

Using {$CLEO} directive implies {$use cleo} so no changes needed for existing scripts.

The above is only necessary for the scripter. A compiled script will continue depend on the presence of CLEO plugins on user's machine so it's a responsibility of the script author to announce them. However in the future we could integrate some kind of dependency check in the CLEO itself, so it reject scripts if required plugins are missing, to avoid game crashes.

Please let me know your thoughts on this feature.

[x87]

We could also agree on some implicit way of adding extensions, for example, have a subfolder inside edit mode directory named extensions where you create a new folder, say, cleo+ and add new files here:

- Sanny Builder\
-- data\
--- sa\
---- extensions\
----- cleo+\
       |- opcodes.txt
       |- templates.txt
       |- keywords.txt
       |- opcodes.ini

Sanny would automatically register a new extension as if it was defined explicitly in the modes.xml