The coal mine

A personal blog for devtime stories.

Site maintained by Ayowel RSS

Improving uploads to Itch.io in GitHub actions

Explaining the reasoning behind the creation of Ayowel/butler-to-itch

Ever since I started working on Ren’Py projects and automating their build and deployment, I have had issues with the actions available to push files to Itch.io using Butler. Those issues are many-fold:

To illustrate this, we can consider the last release of Colors at the time of writing. The total action runtime is 1m50s, the total upload-related runtime is 43s, more than a third of the total runtime. Additionally, you can see under this block of text the part of the script used for the release. It’s a wall and a nightmare to maintain, with a lot of argument redundancy between each step. It’s also very fickle as the file’s path has to match the actual distributed file, which changes with each game version number in RenPy1.

    - name: Push windows version to itch.io
      uses: manleydev/butler-publish-itchio-action@v1.0.3
      env:
        BUTLER_CREDENTIALS: ${{ secrets.BUTLER_CREDENTIALS }}
        ITCH_GAME: colors
        ITCH_USER: kiminako
        CHANNEL: win
        PACKAGE: target/Colors-${{ inputs.version_number }}-win.zip
        VERSION: ${{ inputs.version_number }}
    - name: Push linux version to itch.io
      uses: manleydev/butler-publish-itchio-action@v1.0.3
      env:
        BUTLER_CREDENTIALS: ${{ secrets.BUTLER_CREDENTIALS }}
        ITCH_GAME: colors
        ITCH_USER: kiminako
        CHANNEL: linux
        PACKAGE: target/Colors-${{ inputs.version_number }}-linux.tar.bz2
        VERSION: ${{ inputs.version_number }}
    - name: Push mac version to itch.io
      uses: manleydev/butler-publish-itchio-action@v1.0.3
      env:
        BUTLER_CREDENTIALS: ${{ secrets.BUTLER_CREDENTIALS }}
        ITCH_GAME: colors
        ITCH_USER: kiminako
        CHANNEL: mac
        PACKAGE: target/Colors-${{ inputs.version_number }}-mac.zip
        VERSION: ${{ inputs.version_number }}

Checking the existing actions

All existing actions on the marketplace that push to Itch use the same strategy: Download butler into a small container and add a thin env-based wrapper script to exploit it. This strategy is systematically implemented in a way that does not allow to upload multiple files in a single step and adds an initialisation step at the start of each release run.

Honorable mentions:

As all actions that push rely on the exact same mecanisms to perform the action, using one instead of the other can’t provide significant improvements. Here is the detail of the runtime of the release of Colors mentionned in the introduction:

Label Duration
Set up job 2s
Build yeslayla/butler-publish-itchio-action@v1.0.3 22s
Run actions/checkout@v3 10s
Get Ren’Py from cache2 1s
Build distribution 50s
Push windows version to itch.io 9s
Push linux version to itch.io 6s
Push mac version to itch.io 6s
Other steps 4s
Total 110s

Figuring out a solution

Setting requirements

No available action felt satisfactory, so I considered building my own. It needed to provide at least one of the following without worsening the other:

Picking an implementation medium

There are three options to implement GitHub actions: as a containerized script, as a Javascript project, or as a composite action. Composite actions might help with the first requirement but would forgo the second as the implementation of a loop would become particularly complicated. Containers guaranty a better compatibility accross platforms but add a set-up overhead, while Javascript usually has a smaller overhead but forces to handle platform discrepancies in code.

In this case, most of the overhead is in the set-up, and moving the download to a step handled by a Javascript project would allow to cache the downloaded Butler executable’s file for quick retrieval in later releases3 while removing the cost of the creation of a Docker image and making the implementation of more advanced file handling easier.

Conception

I wanted to have the ability to independantly install butler if needed in a specific script4 and avoid needlessly re-downloading it, should the action be used more than once5. The downloaded Butler executable had to change depending on the current platform and the desired Butler version6, and its installation directory could later be added to the PATH for later commands. Though it is not implemented in the current release, Itch also provides signature files that could be used to verify the downloaded Butler archive in future versions.

Regarding the files pushed to Itch, the main issue that caused redundancy in the configuration was the need to associate channels to pushed files to ensure the proper upstream files were updated.

The following is the state diagram I had when I started writing the implementation:

--- title: Processing logic --- stateDiagram %% diagram for the install action parse_inputs: Parse & validate inputs [*] --> parse_inputs exe_check: Search Butler executable parse_inputs --> exe_check exe_check --> install: not found %% Base url: https://broth.itch.ovh/butler/linux-amd64/LATEST/archive/default state install { [*] --> generate_url generate_url --> download_zip download_zip --> download_signature: check_signature == true download_signature --> verify_signature download_zip --> unpack_butler verify_signature --> unpack_butler unpack_butler --> update_path: update_path == true update_path --> [*] unpack_butler --> [*] } install --> action_check action_check: Check action exe_check --> action_check: found action_check --> [*]: install action_check --> push: push state push { [*] --> list_files list_files --> associate_file_channels associate_file_channels --> push_file push_file --> push_file: Repeat for each file push_file --> [*] } push --> [*]

My next step was to determine what my user requirements were:

GitHub inputs have to be strings. This limitation is not a big issue when the code expects a boolean or a number as the conversion is pretty straightforward, however more complex structures like arrays need specific conventions (actions sometimes use the JSON format to support specific input data). In this case, considering that channels can only be mapped to one file at most and have a limited character set, I decided that users would declare them in the same parameter with the following rule:

This resulted in a multiline string of channel and file patterns that could be mapped to an array of channel+file tuples.

Hence, the configuration requirements could transparently be translated into option classes:

--- title: Mapped input objects --- classDiagram class CommandOptions { +String action +String install_dir } CommandOptions "1" --> "0..1" CommandInstallOptions: install_opt CommandOptions "1" --> "0..1" CommandPushOptions: push_opt class CommandInstallOptions { +Boolean check_signature +Boolean update_path +String butler_version } class CommandPushOptions { +String butler_key +String itch_user +String itch_game +String version +String[][] files +Boolean auto_channel }

Implementation

The implementation was uneventful thanks to the NPM packages in GitHub’s action toolkit:

The only implementation detail of note is the generation of the file/channel association, as the user might not provide one and the script attempts to build one in such a case and raises an error if more that one file is going to be pushed on a single channel:

stateDiagram classDef error fill:red class throw_fnf error class throw_tmf error throw_fnf: Throw File not found throw_tmf: Throw Too many files on channel state preparation { untested_pattern: Has unresolved file pattern pop_pattern: Pop a pattern glob_pattern: Resolve pattern inc_channel_array: Add to channel's file array } [*] --> untested_pattern untested_pattern --> pop_pattern: yes pop_pattern --> glob_pattern glob_pattern --> throw_fnf: =0 match glob_pattern --> inc_channel_array: >0 match inc_channel_array --> untested_pattern state verification { untested_channel: Has unverified channel/files groups? pop_cf_pair: Pop channel/file pair has_null_channel: Has null channel? parse_file_name: Parse file name for channel channel_file_exists: Channel already bound to a file? is_null_channel: Channel is null? raise_warning: Log warning add_file_to_channel: Add file to channel } untested_pattern --> untested_channel: no untested_channel --> pop_cf_pair: yes pop_cf_pair --> has_null_channel has_null_channel --> parse_file_name: yes has_null_channel --> channel_file_exists: no parse_file_name --> channel_file_exists channel_file_exists --> is_null_channel: yes channel_file_exists --> add_file_to_channel: no is_null_channel --> throw_tmf: yes is_null_channel --> raise_warning: no raise_warning --> add_file_to_channel add_file_to_channel --> untested_channel state upload { unpushed_file: Has unpushed file? push_file: Push file } untested_channel --> unpushed_file: no unpushed_file --> push_file: yes push_file --> unpushed_file unpushed_file --> [*]: no

Results

This new action allowed me to drastically reduce the configuration required to push multiple files to itch. The twenty-seven lines required to push three files have now become height and does not need to change for most projects or when adding release targets. At the same time, the overall time it takes to initialize the step and push all files is now 17 seconds (from 43 seconds) for the same action, mostly thanks to the fact that the action does not waste 20+ seconds at the start of the job to create its docker image.

    - name: Push to itch.io
      uses: Ayowel/butler-to-itch@v1.0.0
      with:
        butler_key: ${{ secrets.BUTLER_CREDENTIALS }}
        itch_game: colors
        itch_user: kiminako
        files: target/*
        version: ${{ inputs.version_number }}

  1. Not setting config.version or setting it to an empty string is an efficient way to make the name of the file generated by Ren’Py more stable. ↩

  2. The same cache is used in an action that lints the code each time a commit is pushed, so the cache is guaranteed to exist at release-time. ↩

  3. The Butler archive weighs 10Mo, so the actual advantage of retrieving the file from cache compared to a direct download is more theoretical that practical. ↩

  4. Beyond the command that allows to push files to Itch, Butler provides several commands to handle and verify files - most of which probably wouldn’t be relevant in an action. ↩

  5. Implementing this differentiation also meant that the script could later evolve to handle more Butler commands if needed without breaking existing scripts. ↩

  6. Pratically, there is pretty much no reason that I’m aware of to ever use an outdated Butler version, but this was an option in some other actions and was close to free to implement as a feature. ↩