Size is not correctly calculated and results in a bootloop for big programs

[vladkorotnev]

Board

ESP32 boards (presumably any)

Device Description

Tested on a devkitC from Taobao and on a bare module from Espressif

Hardware Configuration

Doesn't matter

Version

latest master (checkout manually)

IDE Name

PlatformIO + VSCode

Operating System

Windows 10

Flash frequency

40Mhz

PSRAM enabled

yes

Upload speed

921600

Description

I have a partitions.csv with two 0x180000-byte partitions.

When building, the build pipeline finds the partition size correctly (1,572,864 bytes) and claims that the program fits:

Checking size .pio\build\AKI_K875\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [==        ]  19.7% (used 64588 bytes from 327680 bytes)
Flash: [==========]  99.7% (used 1568765 bytes from 1572864 bytes)
CURRENT: upload_protocol = esptool
Looking for upload port...
Auto-detected: COM4

However the actual file created is 1,575,056 bytes (file size, not size on disk) — that is, 0x180890 bytes, which is much bigger than the 0x180000 byte partition. And so esptool tries to flash exactly that, which leads to the firmware crashing on startup:

...
Flash will be erased from 0x00010000 to 0x00190fff...
...
Compressed 1575056 bytes to 952743...
Writing at 0x00010000... (1 %)
...
Writing at 0x0018fa51... (100 %)
Wrote 1575056 bytes (952743 compressed) at 0x00010000 in 16.4 seconds (effective 768.8 kbit/s)...
Hash of data verified.
...
--- Terminal on COM4 | 115200 8-N-1
ets Jul 29 2019 12:21:46

rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
ets Jul 29 2019 12:21:46

rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:1184
load:0x40078000,len:13232
load:0x40080400,len:3028
entry 0x400805e4
ets Jul 29 2019 12:21:46

... many more of the same thing ...

Which is kind of expected, since the flash was written correctly, but is being mapped into the memory according to the partition table — so part of the program ends up being not available, the code jumps somewhere into the uninitialized RAM, and things explodify.

Why does it think the program is smaller than it actually is? Could this be a Windows-specific issue? (I don't have a Linux or Mac machine to test)

I presumed it was a platformio-specific issue, but they seem to be following this repo closely, so it is occurring here too.

Checking in at the ESP32 forums, it seems that the size tool for checking the ELF program size should not be used, as it doesn't include the ESP app header which is added into firmware.bin which is actually used for flashing.

Sketch

https://github.com/vladkorotnev/plasma-clock/tree/9e8c9c064f9b426d3d3e29dfb94840891712d719 — enable feature flags enough to bring the flash usage to 99.5%+, then try to flash it into the memory

Debug Message

N/A

Other Steps to Reproduce

Using this script between build and flash I can catch such conditions and stop the build, but this is a hacky workaround basically: https://github.com/vladkorotnev/plasma-clock/blob/develop/helper/check-size-correctly.py

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[me-no-dev]

Unfortunately there is nothing we can do about PlatformIO. It's an external tool with it's own support and tooling

[me-no-dev]

We are looking into this issue on our side too, but that will not translate to fix on PlatformIO side

[vladkorotnev]

Yep, I understand, but they refused to fix it on their side to avoid discrepancy between this framework and their variant of it, so if this ends up being solved here I'll go bug them again :-)

[yuvashrikarunakaran]

csv

Name, Type, SubType, Offset, Size

nvs, data, nvs, 0x9000, 0x6000
otadata, data, ota, 0xf000, 0x2000
app0, app, ota_0, 0x10000, 0x180000
app1, app, ota_1, 0x190000, 0x180000
spiffs, data, spiffs, 0x310000, 0x1F0000

bash
esptool.py --chip esp32 image_info build/firmware.bin

ini
build_flags =
-Os
-ffunction-sections
-fdata-sections
-Wl,--gc-sections

ini
extra_scripts = pre:check_bin_size.py

python
Import("env")

def before_upload(source, target, env):
firmware_path = str(source[0])
max_size = 0x180000 # Update this to match your partition size

actual_size = os.path.getsize(firmware_path)
if actual_size > max_size:
    print(f"Error: Firmware size ({actual_size} bytes) exceeds partition size ({max_size} bytes).")
    env.Exit(1)

env.AddPreAction("upload", before_upload)

[vladkorotnev]

@yuvashrikarunakaran if you could please format the code using the code block (<> icon in the toolbar) that would be much appreciated!

Also, your solution checks upon upload, and mine checks after build — the latter might be crucial if you are building in CI and then distributing the firmware via OTA (which is what my project is doing). For only local work, either solution is equally good

[Jason2866]

If a fix is made for ArduinoIDE i will add to pioarduino