i2c-ast2600: change bus recovery return type from u8 to int

ast2600_i2c_recover_bus() returns negative errno values (-EPROTO = -71, -ETIMEDOUT = -110) when bus recovery fails, but its return type is declared as u8.

When ast2600_i2c_master_xfer() assigns the u8 return value to int ret,
the truncated unsigned value is zero-extended instead of sign-extended, turning -EPROTO (-71) into +185 (0xb9) and -ETIMEDOUT (-110) into +146 (0x92). Because non-negative return values from master_xfer indicate the number of i2c_msg structs transferred, i2cdev_ioctl_rdwr() treats the failed transfer as succeeding and returns 185 to userspace, causing i2ctransfer to read past its stack-allocated msgs[] array and crash
with SIGSEGV.

Change the return type of ast2600_i2c_recover_bus() from u8 to int so
negative error codes are preserved.

Google-Bug-Id: 569827897
Change-Id: I1817fb09a303ef1118231eb71e39dcc1d1bdf733
Signed-off-by: Sam Agazaryan <samagazaryan@google.com>
1 file changed